- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
CogRL decomposes LLM agent decision-making into a Subgoal Reasoning Policy (SRP), which proposes the next target state, and an Action Reasoning Policy (ARP), which infers an action through inverse dynamics. SRP is optimized using task-return and state-consistency signals, while ARP is periodically updated through supervised learning on observed transitions. Experiments on ALFWorld, BabyAI, and WebShop demonstrate improvements over the evaluated baselines.
Critical strengths
-
The framework is intuitive and promising. Using observed state transitions to supervise intermediate reasoning provides useful feedback beyond task rewards, and the reported performance gains are encouraging.
-
The paper is generally well organized, and the main components and training procedure are clearly explained.
Critical weaknesses
-
Narrative and parenthetical citations are not consistently distinguished, resulting in awkward sentence constructions that occasionally hinder readability.
-
SRP relies on manually designed, structured state representations. Its applicability remains unclear in environments where observations cannot be readily compressed into a small set of JSON fields, or where relevant state information is incomplete.
-
In the evaluated settings, particularly the WebShop prompts, many target-state changes map directly to predefined actions. This makes it difficult to assess the necessity of a learned ARP and its effectiveness when the inverse mapping is ambiguous or more complex.
-
Experiments cover two Qwen models but no other model families, leaving the generality of the training approach across backbones insufficiently established.
-
The core decomposition into target-state planning and inverse dynamics has precedents in traditional RL and related areas, such as D3G and DePO. While its adaptation to LLM agents is worthwhile, the conceptual novelty appears incremental, and the distinct contribution of the proposed optimization scheme could be better established.
Questions for authors
- Could the authors revise the citation formatting to consistently distinguish narrative citations (\citet) from parenthetical citations (\citep)?
- How would SRP represent and predict states in environments with unstructured or partially observed information? Could the authors provide an additional experiment or worked example illustrating the required state abstraction?
- How does ARP handle cases where a target-state change admits multiple actions or lacks a straightforward action mapping? A comparison with a rule-based executor or a frozen ARP would also help establish the benefit of learning this component.
- Could the authors validate the method on a comparably sized model from another family, such as Llama or Gemma?
2 comments
Thank you for your response and for addressing most of my concerns. My main remaining concern is that the method’s reliance on certain intermediate states may make its effectiveness highly dependent on the specific environment or task. For challenging tasks or those involving long trajectories, the intermediate state space can become extremely large, and some intermediate states may be difficult or even impossible to quantify.
I would appreciate further clarification on how the method handles these challenges. If you could clarify this point, I would be happy to raise my score. Best of luck!
Dear Reviewer tHTm,
Thank you for your comments. We have posted a detailed rebuttal addressing the concerns raised. We would greatly appreciate it if you could review it at your convenience and let us know if any questions remain.