- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper proposes Adaptive Event Policy Gradient (AEPG), a test-time method that revises short spans of dependent tokens within a reasoning trajectory. It uses reward-guided policy gradients to adapt a lightweight controller while keeping the language model and reward model frozen. Experiments on three models and five mathematical reasoning benchmarks show improvements in average accuracy, supported by ablations and analyses of search coverage and inference efficiency.
Critical strengths
- Revising several dependent tokens together is a reasonable idea for correcting reasoning errors. The lightweight controller allows this adaptation while keeping the language model frozen.
Critical weaknesses
-
The novelty appears incremental. The method combines existing reward-guided search and policy-gradient techniques, and its added value over PRM-guided search is not clearly established through discussion or direct comparisons.
-
The computational budgets are not fully matched. AEPG uses additional probe rollouts beyond the final responses, making it difficult to separate the benefits of the method from those of additional search computation.
-
The presentation should be clearer. Figure 1 contains too much text and lacks a clear visual focus, making it difficult to identify the main contribution and understand how the method differs from existing approaches.
Questions for authors
-
What evidence distinguishes the proposed event-level adaptation from conventional reward-guided local search, particularly PRM-guided search?
-
Is there evidence that the selected events correspond to coherent reasoning decisions, beyond showing that local prefix changes can improve subsequent generation?
2 comments
Response to Reviewer zYpU(Response2)
> Response to Question 2: Evidence that selected events correspond to coherent reasoning decisions
On this question, our evidence comes mainly from the event selection mechanism, and not merely from showing that a local prefix change can improve the subsequent generation.
Intuitively, whether a position is worth using as the starting point of an event depends on whether it can have a sufficiently large effect on the subsequent generation. We consider two factors. First, the current position must admit alternative next tokens: if a different token is generated, the subsequent tokens are likely to change as well, and this factor can be measured by entropy. Second, the larger the semantic divergence among the candidate tokens, the more likely they are to lead to completely different subsequent reasoning. For example, if the candidates are “so” and “and”, the subsequent reasoning may still follow the same decision; but if the candidates are “however” and “so”, they are likely to lead to opposite reasoning directions. This factor can be formalized as the semantic dispersion of the candidate tokens. Therefore, at the beginning of the procedure, we use entropy * semantic dispersion of candidates as the criterion to select the starting point of an event, and then use an adaptive stopping rule to determine where the event ends.
This is directly supported by the results and analysis in our ablation study: compared with entropy-only, dispersion-only, first-uncertain, and random pivot selection, the joint criterion achieves the best pooled accuracy. Hence, our evidence is not only that local prefix changes can improve the subsequent generation, but also that the event selection mechanism itself tends to locate decision points that have a large influence on subsequent reasoning and whose candidate directions differ substantially, and the ablation results support the effectiveness of this selection criterion.
Response to Reviewer zYpU(Response1)
> Response to Question 1: Distinguishing event-level adaptation from PRM-guided search
There are several aspects to this question.
First, our problem is instance-level test-time optimization, where the key question is how to use a reward to revise an already generated CoT. We introduce this setting in the third paragraph of the Introduction. Intuitively, we focus on this direction because we believe it has greater potential and a higher ceiling than conventional generate-and-search procedures, and this is also partly supported by the comparison results reported in the instance-level optimization work we cite.
Second, regarding the distinction from PRM-guided search that you are most concerned about, we believe the most direct evidence comes from our ablation study. If we do not update Z, that is, if we do not jointly revise the already generated content, the procedure becomes closer to the reward-guided local search you mention: it still uses a reward to screen local prefixes, but it does not perform a joint policy update over multiple dependent decisions within an existing event. Empirically, our method performs better. We should note that Z=0 is not strictly equivalent to PRM-guided search: the former still uses an outcome reward for local prefix selection, whereas the latter is a search guided by dense step-level rewards. Nevertheless, Z=0 is the closest controlled setting to the “select but do not jointly revise” paradigm, and it therefore isolates the gain from joint revision.
In addition, our event is not a strict PRM step. A PRM step usually corresponds to a relatively complete semantic step boundary, whereas an AEPG event is a short sequence of token decisions determined by a pivot and an adaptive stopping rule, and it does not necessarily align with semantic step boundaries. Figure 4 in the paper is a concrete example: the revised event covers a short local span of decisions around the symmetry analysis, rather than a complete PRM step. Thus, AEPG is not simply renaming a PRM step; it jointly revises a shorter, autoregressively dependent span of token decisions.
Finally, there is also a factual distinction: PRM-guided search typically requires an additional process reward model and scores or prunes each step during search, whereas AEPG does not introduce a PRM and does not perform step-level beam search during generation. Instead, it jointly revises an already generated event.