- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes ARC, a method that groups rollouts by interaction strategy such as clarification, progress updates, and confirmation during training to eliminate cross strategy reward bias from relative advantage estimation. Together with the INTER³ interaction framework and the INTER³-86K dataset, it improves the overall average score by about 5.4 points over standard GRPO, with the largest gains on tool calling benchmarks, though some out of domain reasoning benchmarks show slight degradation. It also improves PPO and DAPO, but to a lesser extent than GRPO, and reduces time to first token from 4.91s to 1.27s.
Critical strengths
1.The problem the paper identifies is practically significant. In many cases, an agent directly answering, asking for clarification, or providing progress updates are all reasonable behaviors. However, standard GRPO places these responses into a single reward-normalized group for comparison, which introduces reward model preference bias toward interaction styles and contaminates the relative advantage estimates. This amounts to optimizing answer style rather than genuinely improving the agent's capabilities.
- The experimental coverage is comprehensive. The method is validated on three RL backbones, namely PPO, DAPO, and GRPO, and evaluated on both in-domain and out-of-domain benchmarks.
Critical weaknesses
1.The training process introduces additional “privileged information” in the form of strategy instructions. It is therefore unclear whether the reported gains come from strategy-conditioned advantage estimation or simply from providing the model with these extra strategy instructions.
2.The strategy distribution is highly imbalanced, with 75.3% Progress Update and only 2.5% Alignment Check. It is unclear whether the overall improvement is mainly driven by the dominant Progress Update strategy. If so, the benefit of strategy-based grouping may be limited. Furthermore, the authors do not appear to have provided an AI use statement.
Questions for authors
The strategy instructions injected during training constitute privileged information. How do you rule out that the gains come from the instructions themselves rather than from strategy-conditioned grouping? I suggest a control with the same instructions present but without strategy-based grouping, where instructions are randomly assigned to rollouts within each group. Only if this control shows no improvement can the gains be attributed to the grouping mechanism.
3 comments
I will keep my original rate
Thank you for the additional experiments. However, comparing the new results with Table 2 in the current manuscript raises further questions. On the same five tool-use metrics, GRPO and GRPO+ARC achieve averages of 33.35 and 41.73, respectively, whereas the guidance-free Mixed-strategy and Same-strategy controls achieve only 31.37 and 35.12. Although the latter demonstrates the benefit of grouping, it remains 6.61 points below the full ARC, while the mixed-strategy control is even 1.98 points below the original GRPO baseline. Therefore, the respective contributions of strategy guidance and strategy-conditioned grouping remain unclear, particularly given the different grouping criteria used in the two settings. I will maintain my original rating of 2.
Thank you for the follow-up. We would like to clarify our earlier experimental settings. To isolate the contribution of strategy-conditioned grouping from strategy guidance, the new control uses a modified training protocol, so its absolute scores should not be interpreted as direct ablations of the Table 2 results.
This control uses a modified diagnostic setup with a shorter training horizon: 64 unconditioned candidates per prompt, post-hoc strategy classification, and retention of one eight-response group. In contrast, the original GRPO and ARC experiments generate eight responses per prompt directly, with ARC providing the annotated strategy instruction during generation. This changes the rollout and selection procedure; therefore, the control scores are not directly comparable with the paper’s original ARC and GRPO results.
The relevant comparison is between the matched same-strategy and mixed-strategy conditions. At the matched 50-step checkpoint, same-strategy grouping improves the five-domain average by +3.75. This establishes that grouping can help without prompt guidance; it does not quantify the respective contributions of guidance and grouping to the original ARC gain.
Given the limited resources available during the rebuttal period, we prioritised this controlled diagnostic experiment to provide initial evidence. We acknowledge that a fully matched ablation would provide a more definitive assessment, and we plan to conduct and include these experiments in the camera-ready version.
We thank the reviewer for highlighting the need to compare heterogeneous interaction strategies. We address the instruction control, the dominant-strategy hypothesis and AI-use disclosure separately.
1. Isolating strategy-conditioned advantage estimation from privileged strategy instructions
In order to isolate the effects of strategy guidance during generation and strategy-homogeneous advantage estimation, we have added a guidance-free mechanism control in which strategy instructions are removed entirely.
In this control, both conditions generate responses from the same original prompts, with no strategy information provided to the model. A frozen classifier assigns strategy labels only after generation.
Both conditions generate $ N=64 $ responses per prompt without strategy instructions, and retain $ m=8 $ responses as either a same-strategy or mixed-strategy group. Labels are used solely for group construction; advantages are then computed within the retained group.
Same-strategy grouping improves the five-domain average by 3.75 points, with gains in four of five domains. As neither condition receives strategy guidance, the difference cannot be attributed to privileged prompt information or strategy-guided exploration. This control directly supports ARC’s proposed mechanism: strategy-homogeneous advantage estimation contributes independently of strategy guidance, without implying that it explains the full ARC gain.
2. Does Progress Update alone explain the benefit?
The dominant-family-only configuration does not reproduce the full-suite performance. We refer to Table 13, which directly compares the following configurations:
Despite Progress Update accounting for 75.3% of the corpus, the full suite substantially exceeds the Progress-only configuration on both agentic benchmarks. Therefore, the model benefits from training on multiple strategies, not only Progress Update.
3. AI use statement
We have added an AI use statement to the revised manuscript in accordance with the journal’s guidelines.