acceptodds
Under review as a conference paper at ICLR 2027

CogRL: Enhancing Cognitive Reasoning of LLM Agents via Reinforcement Learning

Abstract

Recent advances in large language models (LLMs) have enhanced agents’ reasoning abilities, but deploying them in dynamic environments remains challenging due to misalignment between internal reasoning and environmental dynamics. Existing RL-based methods often couple reasoning with action and rely only on sparse task rewards, leading to ungrounded reasoning and poor sample efficiency—especially in multi-round interactions. We propose CogRL, an integrated RL framework for enhancing two-stage cognitive reasoning. CogRL makes reasoning outcomes verifiable through a Subgoal Reasoning Policy (SRP) that predicts intended next states, supported by an Action Reasoning Policy (ARP) that generates actions to realize these transitions. Its integrated policy optimization combines separately normalized task-return and state-consistency advantages to train the SRP, enabling learning from unsuccessful trajectories even when task rewards are uninformative. The ARP is trained using periodic inverse-dynamics to support stable online learning. Experiments across multiple interactive environments demonstrate improved task success and data efficiency over strong baselines, with further analysis showing reduced state hallucinations. Our code is available at \url{https://anonymous.4open.science/r/CogRL

Primary area:
reinforcement learning
Keywords:
large language model, reasoning, reinforcement learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_tHTmOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

CogRL decomposes LLM agent decision-making into a Subgoal Reasoning Policy (SRP), which proposes the next target state, and an Action Reasoning Policy (ARP), which infers an action through inverse dynamics. SRP is optimized using task-return and state-consistency signals, while ARP is periodically updated through supervised learning on observed transitions. Experiments on ALFWorld, BabyAI, and WebShop demonstrate improvements over the evaluated baselines.

Critical strengths
  1. The framework is intuitive and promising. Using observed state transitions to supervise intermediate reasoning provides useful feedback beyond task rewards, and the reported performance gains are encouraging.

  2. The paper is generally well organized, and the main components and training procedure are clearly explained.

Critical weaknesses
  1. Narrative and parenthetical citations are not consistently distinguished, resulting in awkward sentence constructions that occasionally hinder readability.

  2. SRP relies on manually designed, structured state representations. Its applicability remains unclear in environments where observations cannot be readily compressed into a small set of JSON fields, or where relevant state information is incomplete.

  3. In the evaluated settings, particularly the WebShop prompts, many target-state changes map directly to predefined actions. This makes it difficult to assess the necessity of a learned ARP and its effectiveness when the inverse mapping is ambiguous or more complex.

  4. Experiments cover two Qwen models but no other model families, leaving the generality of the training approach across backbones insufficiently established.

  5. The core decomposition into target-state planning and inverse dynamics has precedents in traditional RL and related areas, such as D3G and DePO. While its adaptation to LLM agents is worthwhile, the conceptual novelty appears incremental, and the distinct contribution of the proposed optimization scheme could be better established.

Questions for authors
  1. Could the authors revise the citation formatting to consistently distinguish narrative citations (\citet) from parenthetical citations (\citep)?
  2. How would SRP represent and predict states in environments with unstructured or partially observed information? Could the authors provide an additional experiment or worked example illustrating the required state abstraction?
  3. How does ARP handle cases where a target-state change admits multiple actions or lacks a straightforward action mapping? A comparison with a rule-based executor or a frozen ARP would also help establish the benefit of learning this component.
  4. Could the authors validate the method on a comparably sized model from another family, such as Llama or Gemma?
2 comments

Dear Reviewer tHTm,

Thank you for your comments. We have posted a detailed rebuttal addressing the concerns raised. We would greatly appreciate it if you could review it at your convenience and let us know if any questions remain.

Thank you for your response and for addressing most of my concerns. My main remaining concern is that the method’s reliance on certain intermediate states may make its effectiveness highly dependent on the specific environment or task. For challenging tasks or those involving long trajectories, the intermediate state space can become extremely large, and some intermediate states may be difficult or even impossible to quantify.

I would appreciate further clarification on how the method handles these challenges. If you could clarify this point, I would be happy to raise my score. Best of luck!

Reviewer_Zo7aOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

CogRL separates agent decisions into a Subgoal Reasoning Policy (SRP), which predicts the next environment state, and an Action Reasoning Policy (ARP), which chooses an action to realize that state. It trains the SRP with separately normalized task-return and state-consistency advantages, and periodically fine-tunes the ARP on observed transitions. Across ALFWorld, BabyAI, and WebShop, the paper reports higher task performance than the evaluated RL baselines, along with fewer transition-level prediction errors.

Critical strengths
  • The paper identifies a real limitation of multi-turn RL for LLM agents — reasoning optimized only through sparse task reward — and turns it into a concrete, checkable intermediate prediction.
  • Extending group-based RL from trajectories to decision states yields step-level advantage estimation without a critic, and the consistency signal lets the method learn from failed rollouts.
Critical weaknesses
  • The decoupled architecture is not isolated by the ablation. The reported ablations vary only the consistency weight and aggregation level; no arm combines an entangled single policy with the same consistency signal, removes periodic ARP SFT, or replaces reference-state grouping with trajectory-level GRPO grouping. The headline gains therefore cannot be attributed to decoupling rather than the consistency reward, grouping, the second LoRA adapter, the ARP SFT data, or prompt differences.
  • The reported results use three random seeds, without a reported significance analysis. This is especially relevant to the WebShop success-rate gains over GiGPO, alongside the reported standard deviations. More runs or an appropriate statistical comparison would clarify how robust these margins are.
Questions for authors

Please see the above comments.

4 comments

Dear Reviewer Zo7a,

Thank you for your comments. We have posted a detailed rebuttal addressing the concerns raised. We would greatly appreciate it if you could review it at your convenience and let us know if any questions remain.

Dear Reviewer Zo7a,

Thank you again for your time and thoughtful feedback on our paper. As the author-reviewer discussion period is drawing to a close, we wanted to gently follow up in case you have had a chance to review our rebuttal and the newly added experiments.

We completely understand how busy this period can be. If any questions or concerns remain before the discussion period concludes, we would be very grateful for the opportunity to clarify them.

Thanks for the authors' detailed responses and additional experiments. I will raise my score.

Thank you for carefully considering our responses and revising your score upward. Your constructive feedback has helped us strengthen the empirical validation and improve the clarity of the paper. We sincerely appreciate your time and thoughtful review.

Reviewer_2FPMOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes CogRL, which separates decision making into a Subgoal Reasoning Policy and an Action Reasoning Policy with separate LoRA adapters. The former predicts the intended next state and learns from task returns and state-consistency feedback. The latter generates an action and is trained through inverse-dynamics supervised fine-tuning. Experiments use two Qwen backbones on ALFWorld, BabyAI, and WebShop.

Critical strengths

Separating SRP and ARP allows subgoal reasoning and action generation to use different training objectives. Predicted next states can be checked against observed transitions, providing feedback even when trajectories fail. CogRL improves overall task success over GiGPO across three environments and two backbones. The ALFWorld comparison under a matched training-time budget also shows an improvement.

Critical weaknesses

The current ablations do not isolate the benefit of separating SRP and ARP. They examine the state-consistency weight and advantage versus reward summation, but do not compare the two-policy design with a single policy under comparable supervision and training budgets.

Questions for authors

How does CogRL compare with a single policy that generates both the intended next state and the action under comparable supervision and training budgets?

1 comment

Dear Reviewer 2FPM,

Thank you for your comments. We have posted a detailed rebuttal addressing the concerns raised. We would greatly appreciate it if you could review it at your convenience and let us know if any questions remain.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.