- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
Recent language models are limited to a fixed set of reasoning modes and lack adaptive reasoning. The paper proposes an adaptive reasoning system for text-based interactive environments. The paper decomposes cognitive procedures into four types, Ground, Plan, Check, and Act, and uses a grammar to constrain dependencies between procedures. The proposed method, EvoCog, is three-fold: (1) The model is trained on a supervised dataset to initialize the cognitive program; (2) The RL training stage, Sparse Counterfactual Program Optimization (SCPO), learns two things at the same time: how to select a correct cognitive program, and tries to reduce the overall cost; (3) The language model agent can revise the program. EvoCog is evaluated in three text-based interactive environments and performs among the best. The qualitative results show the LLM agent learns to use different cognitive programs.
Critical strengths
- Adaptive reasoning procedure is a useful skill for language models; the paper uses RL to learn an adaptive cognitive program to solve the problem in the text-based interactive environment.
- EvoCog shows that the cognitive patterns shift from earlier to later stages. The method learns adaptive reasoning routing across stages.
- EvoCog uses the fewest tokens and performs among the best.
Critical weaknesses
- The paper's writing is not informative enough to fully understand the method. Some details are missing, making the paper hard to reproduce. It is also hard to evaluate each component's contribution.
- One of the baselines, CogRouter [1], also shows good performance in all environments, and the authors do not provide statistical evidence to show that the proposed method is better. Further qualitative analysis comparing EvoCog and CogRouter may be needed to distinguish the proposed method's strengths and weaknesses. [1] Yang, Ruihan, et al. "Think Fast and Slow: Step-Level Cognitive Depth Adaptation for LLM Agents." arXiv preprint arXiv:2602.12662 (2026).
Questions for authors
- The paper does not provide enough information about the method. In Section 3.4, you introduce the loss but do not define it. What does a router look like? Is it a neural network or a retrieval mechanism?
- The best-performing baseline is CogRouter [1]; is your method statistically better? In Section 4.2 and Table 1, EvoCog wins on one success-rate metric, ties on one, and loses on one, so it is hard to say your method is significantly better than CogRouter [1].
- In Section 3.3, what is the teacher model used to construct the supervised cognitive signal?
- In RL training, how is the challenger program (the RL baseline) selected?
- To my knowledge, the CogRouter [1] also uses RL training; why is it classified as an adaptive reasoning method in Section 4? And what are the main differences between your method and CogRouter [1]? Does your method use the same training data?
- What is the token usage with the external diagnosis module? Does the changed cognitive program use more tokens?
5 comments
Response to Weekness2 & Qestion2, 5
W #2 & Q #2: Performance–cost comparison
As shown in Table 1, EvoCog achieves a higher average success rate than CogRouter (77.47% vs. 75.50%) while reducing generated tokens by 53.8%. In particular, EvoCog matches CogRouter's success rate on ALFWorld with 69.1% fewer tokens, improves success on ScienceWorld with 38.0% fewer tokens, and achieves comparable success on WebShop with 52.4% fewer tokens.
To address the concern about statistical reliability, we additionally conducted multi-seed experiments for EvoCog in Table1. Table 1: Multi-seed evaluation of EvoCog on Qwen2.5-7B-Instruct. SR denotes success rate (%), and token counts are averaged per task.
| Benchmark | Seed 42 SR | Seed 43 SR | Seed 44 SR | SR (Mean ± Std) | Output Tokens (Mean ± Std) |
|---|---|---|---|---|---|
| ALFWorld | 92.5 | 93.5 | 92.0 | 92.67 ± 0.76 | 562.68 ± 34.41 |
| ScienceWorld | 76.5 | 73.0 | 75.5 | 75.00 ± 1.80 | 981.42 ± 53.58 |
| WebShop | 61.4 | 62.8 | 63.2 | 62.47 ± 0.95 | 876.49 ± 47.09 |
Q #5. Key differences from CogRouter.
Both methods employ reinforcement learning. We categorize CogRouter as an adaptive reasoning method based on its research objective rather than its training algorithm. The key differences are:
- Cognitive representation: CogRouter adapts reasoning depth through four predefined cognitive levels, whereas EvoCog adapts reasoning structure through composable programs with different operation types and ordering.
- Credit assignment: CogRouter's CoPO expands successful trajectories across four cognitive levels and reweights advantages based on action prediction confidence. In contrast, SCPO evaluates only one alternative program at selected failure-sensitive states through a suffix rollout, assigning counterfactual credit based on actual downstream utility rather than action confidence.
- Efficiency: This sparse design avoids exhaustive cognitive-program comparisons. On Qwen2.5-7B, EvoCog achieves 53.8% fewer inference tokens with a higher average success rate (77.47% vs. 75.50%). Moreover, the reported RL training times are shorter:.The reported RL training times are 36.5h vs. 55.0h on ALFWorld and 48.4h vs. 86.3h on ScienceWorld (EvoCog vs. CogRouter, respectively).
Training data: Both methods use benchmark training partitions with held-out evaluation tasks. However, their supervision differs: EvoCog constructs program-conditioned traces from AgentTraj-L expert trajectories, whereas CogRouter uses cognitive-level-conditioned supervision. We match the evaluation splits and rollout budgets wherever applicable and will clarify the data construction differences in the revision.
Response to Q6
Q #6. Token Overhead of Test-Time Cognitive Evolution
We acknowledge that test-time cognitive evolution introduces additional token consumption and API costs. As shown in Table 1 below, the GPT-4o-based diagnosis and repair modules consume 40,275, 194,580, and 38,683 tokens in total on ALFWorld, ScienceWorld, and WebShop, respectively.
Table 1: Total token consumption of external diagnosis and repair modules.
| Benchmark | External Module Tokens |
|---|---|
| ALFWorld | 40,275 |
| ScienceWorld | 194,580 |
| WebShop | 38,683 |
However, these adaptation costs should be distinguished from subsequent execution costs. As shown in Table 2, refined programs do not consistently increase execution-token consumption: tokens per task decrease by 1.58% on ALFWorld, increase by 9.07% on ScienceWorld, and remain nearly unchanged on WebShop (+0.11%).
Table 2: Execution-token consumption before and after cognitive program refinement.
| Benchmark | Before (Tokens/Task) | After (Tokens/Task) | Change (%) |
|---|---|---|---|
| ALFWorld | 537.8 | 529.3 | −1.58 |
| ScienceWorld | 957.1 | 1043.9 | +9.07 |
| WebShop | 820.3 | 821.2 | +0.11 |
These results indicate that cognitive refinement incurs additional adaptation costs but does not necessarily increase subsequent execution-token usage. Moreover, verified refinements are stored in the external cognitive library for reuse, without requiring repeated repair for every subsequent task.
Response for weakness 1 & question 1, 3, 4
We sincerely thank the reviewer for the thoughtful and constructive feedback. We appreciate your recognition of EvoCog's adaptive cognitive routing and its ability to learn different reasoning patterns across task stages.
The Router is a trainable language-model-based policy, not a retrieval mechanism. It shares the backbone with the Executor but serves a distinct function: selecting a grammar-valid cognitive program based on the current state, while the Executor generates the corresponding reasoning trace and action. As detailed in Appendix A.2.3, the Router is optimized using a PPO-style loss with group-normalized, cost-aware trajectory advantages. SCPO additionally introduces counterfactual preference supervision for program selection.
The teacher is GPT-4o, as specified in Appendix A.2.1. It generates program-conditioned reasoning traces while preserving expert actions, without providing optimal program-selection labels.
SCPO employs a two-step procedure to construct counterfactual comparisons. First, failure-sensitive anchor selection. Rather than branching at every interaction step, SCPO identifies informative anchor states using three criteria:
Second, challenger program selection. At each anchor state, SCPO selects one challenger from the remaining grammar-valid programs, excluding the incumbent selected in the original rollout. The challenger is evaluated through an additional suffix rollout from the same state, and the downstream utility difference provides counterfactual supervision for program selection.
We clarify that the contributions of SCPO's two key components are evaluated through controlled ablations in Table 3. Removing counterfactual program credit reduces success rates by 9.0, 11.5, and 5.8 percentage points on ALFWorld, ScienceWorld, and WebShop, respectively. Removing the cost-aware objective increases token consumption by approximately 1.9–3.2× across the three environments.
These results demonstrate that counterfactual credit primarily improves program selection, while cost-aware optimization reduces inference cost. We will further clarify the Router architecture, challenger sampling procedure, and implementation details in the revised manuscript.