- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies credit assignment for agentic RL. The authors propose ASCT, which runs training-time tree search over alternative legal actions from recoverable prefixes, estimates local action values, and centers them with the frozen actor's probabilities to credit PPO updates on actor-sampled trajectories. Uniform, UCT, and AgentUCT instantiate the search. On HotpotQA RAG with Qwen3-4B, all three evaluators improve held-out utility over trajectory-return PPO and VinePPO, and ASCT-AgentUCT reaches 0.6187 versus 0.5939 for VinePPO.
Critical strengths
- Credit assignment for local decisions in multi-step agentic workflows is an important problem, and using search only at training time while deploying the actor alone is a reasonable design.
- The paper reports auxiliary token cost, difficulty splits, transfer results, and an ActorRollout control, which provides useful diagnostics on the proposed credit estimator.
Critical weaknesses
- This paper seems largely written by an LLM, which makes it really hard for humans to follow the motivation and logic across sentences. Thus, I think the manuscript needs substantial rewriting to meet the bar of publication at ICLR.
- The novelty seems limited. I cannot identify a principled rationale in this workflow, which reads like a combination of several established agentic tree search methods [1-6]. I think the authors need a clear discussion and comparison to justify the novelty over them.
[1] VinePPO: Refining Credit Assignment in RL Training of LLMs
[2] Language Agent Tree Search Unifies Reasoning, Acting, and Planning in Language Models
[3] Tree Search for LLM Agent Reinforcement Learning
[4] Branching Policy Optimization: Sandbox-Native Language Agent Reinforcement Learning
[5] Agent-UCT: Upper Confidence Bounds Applied to Trees for Agentic Workflow Optimization with Cost-Awareness
[6] TRACE: A Unified Rollout Budget Allocation Framework for Efficient Agentic Reinforcement Learning
Questions for authors
- Can the authors add a clear discussion of the novelty over [1-6], especially [5] (whose search rules are directly inherited) and [6]?
- Uniform achieves the highest F1, while AgentUCT mainly wins on execution cost. Does this mean the credit interface, rather than the search allocator, is the main source of gain?
- The AT2PO adaptation performs close to PPO without method-specific tuning. Is this comparison fair, and if so, why is learning from branch collections not helpful here?
3 comments
Thanks for the response, but the two critical weaknesses are still not addressed. The writing is still poor: many sentences and paragraphs lack logical connection, and there are still stray fragments like code paths left in the text, making the current manuscript far below publication standard. Not just the abstract and introduction, the whole paper needs a rewrite. Partly because of the writing, I still find it hard to see the principled rationale and novelty behind this workflow.
Thank you for your response and further feedback.
-
Code paths: The referenced paths are intentional implementation references in Appendix A.3, included for reproducibility. This is a common practice in ML papers, including previously accepted ICLR papers.
-
AI-assisted writing: We have explicitly disclosed our use of AI assistance in accordance with ICLR 2027 policy and take full responsibility for the manuscript.
We appreciate your time and consideration.
Thank you for the constructive feedback.
1. Novelty, relationship to prior work, and writing
There are several approaches to multi-step exploration and credit assignment in agentic RL:
In ASCT, the Actor generates trajectories for PPO updates, while a separate tree-search evaluator estimates counterfactual action values. This separates two interacting problems: Policy Optimization and Search-based Value Evaluation.
Here, tree-search selection and budget allocation need not follow the LLM Actor's policy. Instead, strategies such as Uniform, UCT, and AgentUCT can guide tree expansion and construct the Q-value table used for credit assignment. This allows evaluation to cover alternatives the actor rarely selects while keeping policy-update trajectories actor-sampled. Even a simple Monte Carlo evaluator can still benefit from this separation.
Regarding [5], [6], and [2], AgentUCT provides a search and allocation strategy, like UCT or Uniform. TRACE focuses on fine-tuning a Qwen3-0.6B predictor to allocate additional searches during GRPO training. LATS is an agent framework without training LLMs or updating LLMs’ parameters. These methods address different problems, and we thank the reviewer for highlighting the need to clarify this distinction.
We have rewritten the Abstract and Introduction in the latest version to clarify our novelty and relationship to prior work. We hope this addresses the reviewer's concerns and welcome any further questions.
2. Uniform versus AgentUCT
We agree with the reviewer's observation. Uniform achieves the highest mean F1 (0.7316), while AgentUCT achieves the highest mean utility (0.6187) with lower evaluation cost. Their performance differences are relatively small.
This is precisely one of the questions ASCT investigates: different search evaluators can improve the final policy through the same credit interface, while their allocation strategies lead to different trade-offs between policy quality and computational cost.
We do not claim that AgentUCT consistently outperforms Uniform, and ASCT does not depend on a particular search allocation algorithm.
In an additional full-policy control experiment, Root MC balances root actions and samples later actions uniformly, without internal tree visit allocation. It achieves utility of 0.6159±0.0125, compared with ActorRollout’s 0.5975±0.0128, and outperforms ActorRollout in all three seeds. These results show that a simple evaluator can provide useful training credit without following the actor’s continuation policy.
3. AT²PO comparison
Our AT²PO implementation uses the same workflow and experimental setup as the other methods, without additional SFT or method-specific tuning, to maintain comparable experimental conditions.
We cannot determine exactly why its performance is close to PPO. Method-specific tuning or the absence of additional SFT could affect the results, but we have not evaluated these possibilities.
Importantly, we do not claim that branch-based policy optimization is generally ineffective.
We have explicitly clarified the absence of additional SFT as an experimental control and the scope of our conclusions in the revised manuscript to avoid misunderstanding.
Thank you again for the suggestions. We hope these clarifications address the main concerns, and we welcome any further questions.