acceptodds
Under review as a conference paper at ICLR 2027

EvoCog: Composable Cognitive Programs for Adaptive Reasoning in LLM Agents

Abstract

Large language-model-based agents have demonstrated remarkable capabilities in complex decision-making and tool-use tasks. However, existing adaptive reasoning methods typically adapt computation within a predefined set of reasoning modes, limiting their ability to optimize and evolve the structured cognitive procedures required across diverse interactive environments. In this paper, we introduce EvoCog, a framework that represents an agent’s cognitive space as executable programs built from four typed operations: Ground, Plan, Check and Act. A causal grammar constrains dependencies among these operations, defining a space of structurally valid programs. EvoCog learns and evolves these programs through a three-stage pipeline. First, program-conditioned supervised fine-tuning teaches a shared executor to follow assigned programs while preserving expert actions. Second, Sparse Counterfactual Program Optimization (SCPO) optimizes task execution and a cost-aware router for step-level program selection. Finally, EvoCog supports test-time adaptation by iteratively repairing its cognitive programs from interaction failures, without updating model parameters. Experiments on ALFWorld, ScienceWorld and WebShop demonstrate that EvoCog improves interactive task solving while reducing inference cost. With Qwen2.5-7B, it reaches a 77.47% success rate, outperforming GPT-5(+26.57%) and GRPO(+15.1%), while using 51.4% fewer tokens.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM agents, Large Language Models, Reasoning, Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_vQV4OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Recent language models are limited to a fixed set of reasoning modes and lack adaptive reasoning. The paper proposes an adaptive reasoning system for text-based interactive environments. The paper decomposes cognitive procedures into four types, Ground, Plan, Check, and Act, and uses a grammar to constrain dependencies between procedures. The proposed method, EvoCog, is three-fold: (1) The model is trained on a supervised dataset to initialize the cognitive program; (2) The RL training stage, Sparse Counterfactual Program Optimization (SCPO), learns two things at the same time: how to select a correct cognitive program, and tries to reduce the overall cost; (3) The language model agent can revise the program. EvoCog is evaluated in three text-based interactive environments and performs among the best. The qualitative results show the LLM agent learns to use different cognitive programs.

Critical strengths
  1. Adaptive reasoning procedure is a useful skill for language models; the paper uses RL to learn an adaptive cognitive program to solve the problem in the text-based interactive environment.
  2. EvoCog shows that the cognitive patterns shift from earlier to later stages. The method learns adaptive reasoning routing across stages.
  3. EvoCog uses the fewest tokens and performs among the best.
Critical weaknesses
  1. The paper's writing is not informative enough to fully understand the method. Some details are missing, making the paper hard to reproduce. It is also hard to evaluate each component's contribution.
  2. One of the baselines, CogRouter [1], also shows good performance in all environments, and the authors do not provide statistical evidence to show that the proposed method is better. Further qualitative analysis comparing EvoCog and CogRouter may be needed to distinguish the proposed method's strengths and weaknesses. [1] Yang, Ruihan, et al. "Think Fast and Slow: Step-Level Cognitive Depth Adaptation for LLM Agents." arXiv preprint arXiv:2602.12662 (2026).
Questions for authors
  1. The paper does not provide enough information about the method. In Section 3.4, you introduce the loss Lrouter{L_{router}} but do not define it. What does a router look like? Is it a neural network or a retrieval mechanism?
  2. The best-performing baseline is CogRouter [1]; is your method statistically better? In Section 4.2 and Table 1, EvoCog wins on one success-rate metric, ties on one, and loses on one, so it is hard to say your method is significantly better than CogRouter [1].
  3. In Section 3.3, what is the teacher model used to construct the supervised cognitive signal?
  4. In RL training, how is the challenger program (the RL baseline) selected?
  5. To my knowledge, the CogRouter [1] also uses RL training; why is it classified as an adaptive reasoning method in Section 4? And what are the main differences between your method and CogRouter [1]? Does your method use the same training data?
  6. What is the token usage with the external diagnosis module? Does the changed cognitive program use more tokens?
5 comments
AuthorsOpenReview ↗

Response for weakness 1 & question 1, 3, 4

We sincerely thank the reviewer for the thoughtful and constructive feedback. We appreciate your recognition of EvoCog's adaptive cognitive routing and its ability to learn different reasoning patterns across task stages.


Q #1. Router architecture and optimization

The Router is a trainable language-model-based policy, not a retrieval mechanism. It shares the backbone with the Executor but serves a distinct function: selecting a grammar-valid cognitive program based on the current state, while the Executor generates the corresponding reasoning trace and action. As detailed in Appendix A.2.3, the Router is optimized using a PPO-style loss with group-normalized, cost-aware trajectory advantages. SCPO additionally introduces counterfactual preference supervision for program selection.

Q #3. Teacher model

The teacher is GPT-4o, as specified in Appendix A.2.1. It generates program-conditioned reasoning traces while preserving expert actions, without providing optimal program-selection labels.

Q #4. Challenger program selection

SCPO employs a two-step procedure to construct counterfactual comparisons. First, failure-sensitive anchor selection. Rather than branching at every interaction step, SCPO identifies informative anchor states using three criteria:

  • Divergence: The first action divergence between successful and failed trajectories sharing a common state prefix.
  • Recovery: Invalid actions, explicit failure feedback, or nonterminal transitions without observable progress.
  • Stagnation: Repeated state–action pairs, no progress for three consecutive steps.

Second, challenger program selection. At each anchor state, SCPO selects one challenger from the remaining grammar-valid programs, excluding the incumbent selected in the original rollout. The challenger is evaluated through an additional suffix rollout from the same state, and the downstream utility difference provides counterfactual supervision for program selection.

W #1. Component contributions

We clarify that the contributions of SCPO's two key components are evaluated through controlled ablations in Table 3. Removing counterfactual program credit reduces success rates by 9.0, 11.5, and 5.8 percentage points on ALFWorld, ScienceWorld, and WebShop, respectively. Removing the cost-aware objective increases token consumption by approximately 1.9–3.2× across the three environments.

These results demonstrate that counterfactual credit primarily improves program selection, while cost-aware optimization reduces inference cost. We will further clarify the Router architecture, challenger sampling procedure, and implementation details in the revised manuscript.

AuthorsOpenReview ↗

Response to Weekness2 & Qestion2, 5

W #2 & Q #2: Performance–cost comparison

As shown in Table 1, EvoCog achieves a higher average success rate than CogRouter (77.47% vs. 75.50%) while reducing generated tokens by 53.8%. In particular, EvoCog matches CogRouter's success rate on ALFWorld with 69.1% fewer tokens, improves success on ScienceWorld with 38.0% fewer tokens, and achieves comparable success on WebShop with 52.4% fewer tokens.

To address the concern about statistical reliability, we additionally conducted multi-seed experiments for EvoCog in Table1. Table 1: Multi-seed evaluation of EvoCog on Qwen2.5-7B-Instruct. SR denotes success rate (%), and token counts are averaged per task.

BenchmarkSeed 42 SRSeed 43 SRSeed 44 SRSR (Mean ± Std)Output Tokens (Mean ± Std)
ALFWorld92.593.592.092.67 ± 0.76562.68 ± 34.41
ScienceWorld76.573.075.575.00 ± 1.80981.42 ± 53.58
WebShop61.462.863.262.47 ± 0.95876.49 ± 47.09

Q #5. Key differences from CogRouter.

Both methods employ reinforcement learning. We categorize CogRouter as an adaptive reasoning method based on its research objective rather than its training algorithm. The key differences are:

  • Cognitive representation: CogRouter adapts reasoning depth through four predefined cognitive levels, whereas EvoCog adapts reasoning structure through composable programs with different operation types and ordering.
  • Credit assignment: CogRouter's CoPO expands successful trajectories across four cognitive levels and reweights advantages based on action prediction confidence. In contrast, SCPO evaluates only one alternative program at selected failure-sensitive states through a suffix rollout, assigning counterfactual credit based on actual downstream utility rather than action confidence.
  • Efficiency: This sparse design avoids exhaustive cognitive-program comparisons. On Qwen2.5-7B, EvoCog achieves 53.8% fewer inference tokens with a higher average success rate (77.47% vs. 75.50%). Moreover, the reported RL training times are shorter:.The reported RL training times are 36.5h vs. 55.0h on ALFWorld and 48.4h vs. 86.3h on ScienceWorld (EvoCog vs. CogRouter, respectively).

Training data: Both methods use benchmark training partitions with held-out evaluation tasks. However, their supervision differs: EvoCog constructs program-conditioned traces from AgentTraj-L expert trajectories, whereas CogRouter uses cognitive-level-conditioned supervision. We match the evaluation splits and rollout budgets wherever applicable and will clarify the data construction differences in the revision.

AuthorsOpenReview ↗

Response to Q6

Q #6. Token Overhead of Test-Time Cognitive Evolution

We acknowledge that test-time cognitive evolution introduces additional token consumption and API costs. As shown in Table 1 below, the GPT-4o-based diagnosis and repair modules consume 40,275, 194,580, and 38,683 tokens in total on ALFWorld, ScienceWorld, and WebShop, respectively.

Table 1: Total token consumption of external diagnosis and repair modules.

BenchmarkExternal Module Tokens
ALFWorld40,275
ScienceWorld194,580
WebShop38,683

However, these adaptation costs should be distinguished from subsequent execution costs. As shown in Table 2, refined programs do not consistently increase execution-token consumption: tokens per task decrease by 1.58% on ALFWorld, increase by 9.07% on ScienceWorld, and remain nearly unchanged on WebShop (+0.11%).

Table 2: Execution-token consumption before and after cognitive program refinement.

BenchmarkBefore (Tokens/Task)After (Tokens/Task)Change (%)
ALFWorld537.8529.3−1.58
ScienceWorld957.11043.9+9.07
WebShop820.3821.2+0.11

These results indicate that cognitive refinement incurs additional adaptation costs but does not necessarily increase subsequent execution-token usage. Moreover, verified refinements are stored in the external cognitive library for reuse, without requiring repeated repair for every subsequent task.

Reviewer_2HbbOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2(+1)
Presentation
2
Soundness
2(+1)
Read review
Paper summary

For adaptive reasoning, this paper proposes EvoCog, a framework that programs an agent's operations, uses supervised fine-tuning to teach the model how to execute operations, uses reinforcement learning to teach the model how to select which operation to execute, and records the framework's failure experience in an external library. Experimental results show that the framework outperforms related baselines on three evaluation sets.

Critical strengths
  1. The overall research design follows the mainstream recipe: characterizing agent behavior with four atomic operation programs, training operation selection and execution via SFT and RL respectively, and using an external library as an engineering fallback.
Critical weaknesses
  1. Limited novelty. The idea of atomizing agent behavior is not new; related designs already appear in a large body of work. I suggest comparing not only against frontier models and other prompt-based methods, but also horizontally against methods evaluated on the same ALFWorld, ScienceWorld, and WebShop suites, e.g., GLIDER [1] and Self-Generated In-Context Examples [2]. In addition, I suggest running the experiments multiple times, and explaining, for instance, why the EvoCog framework can lift the success rate of a relatively weak model like qwen2.5-7B from 14% to a striking 92.5%, surpassing GPT-5's 78.5%. Is this capability really endowed by the EvoCog framework itself, or is it because, e.g., the teacher model in EvoCog Stage 1 effectively distilled the answers?
  2. Weak reproducibility — or, put differently, insufficiently transparent writing. Key information is missing from the paper, which undermines the transparency and reproducibility of the study, including but not limited to:
    1. the teacher model mentioned at line 213;
    2. what the pre-defined failure-sensitivity criterion at line 249 is;
    3. how the external diagnosis module and the repair module at lines 278 and 281 are constructed;
    4. whether the model is trained on the evaluation environments and could therefore overfit to them;
    5. other key implementation details of the algorithm.

[1] Divide and Conquer: Grounding LLMs as Efficient Decision-Making Agents via Offline Hierarchical Reinforcement Learning (ICML 2025)

[2] Self-Generated In-Context Examples Improve LLM Agents for Sequential Decision-Making Tasks (NeurIPS 2025)

Questions for authors

Points where the exposition caused me confusion:

  1. Line 47: how exactly does Figure 1 illustrate that "programs with similar computational cost may produce drastically different utility"?
  2. Line 48: if cognition itself is represented as executable programs, isn't a program itself a fixed pattern of reasoning? Yet the text states that this is not a fixed reasoning pattern — how should I reconcile these statements?
  3. Please explain the concrete composition of the equation at line 265.
  4. What exactly do P1–P10 in Figure 3 (line 378) denote — the definitions given at line 198? If so, a more explicit definition in the main text would help.
4 comments
AuthorsOpenReview ↗

Response for weakness 1

We sincerely thank the reviewer for the careful evaluation and constructive feedback. We believe that several important aspects of EvoCog were not sufficiently emphasized in the original presentation. We address these concerns below and clarify the relevant methodological details.


(a) Clarifying the novelty of EvoCog.

We agree that decomposing agent behavior into smaller operations is not, by itself, a new idea. However, the main contribution of EvoCog is not the introduction of four atomic operations, but the formulation of adaptive reasoning as the selection, execution, and optimization of composable cognitive programs.

Specifically, EvoCog introduces two key methodological distinctions:

  • Composable cognitive control: Unlike methods that select among predefined, complete reasoning modes, EvoCog represents cognition as typed operations composed under a causal grammar. This allows programs to differ not only in reasoning depth but also in the types and ordering of cognitive operations.
  • Program-level counterfactual optimization: To address the credit assignment challenge in long-horizon tasks, SCPO directly evaluates alternative cognitive programs at matched states through sparse counterfactual rollouts. Unlike CogRouter's confidence-based optimization or GiGPO's step-level advantage estimation, SCPO provides explicit utility-based feedback for cognitive program selection.

Test-time evolution further extends this framework by attaching verified, guarded modifications to existing programs without updating model parameters.

We will revise the introduction and contribution statements to better emphasize these specific contributions and distinguish them from established training and memory mechanisms.

(b) Additional comparisons.

Following the reviewer's suggestion, we have added comparisons with GLIDER and Self-Generated In-Context Examples across three benchmarks.

Table 1: Success rates (%) of additional baselines on Qwen2.5-7B-Instruct.

MethodALFWorldScienceWorldWebShop
GLIDER74.569.556.5
Self-Generated ICL56.046.024.5
EvoCog (Ours)92.576.561.4

(c) Understanding the improvement from 14.0% to 92.5%. As described in Appendix A.2.1, stage I uses GPT-4o to generate program-conditioned reasoning traces while preserving the original expert actions. Importantly, it teaches program execution, not program selection. Table 2 separates the contributions of supervised initialization and subsequent RL optimization:

Table 2: Performance breakdown of EvoCog on ALFWorld using Qwen2.5-7B-Instruct.

MethodALFWorld SR (%)
Direct Qwen2.5-7B14.0
Program-SFT56.5
Program-SFT + GRPO82.5
Program-SFT + GiGPO91.5
Program-SFT + SCPO92.5

Supervised initialization contributes substantially to performance. However, SCPO provides an additional 36.0 percentage-point improvement over Program-SFT and achieves stronger average performance with lower token consumption than GRPO and GiGPO under the same SFT initialization. Moreover, removing counterfactual credit reduces success to 83.5% (Table 3 of the submission). These results demonstrate that the gains cannot be attributed to teacher distillation alone.

We also agree that reporting uncertainty is important. We conducted three independent SCPO training runs with different random seeds. Table 3 reports the individual success rates and the mean ± standard deviation across runs.

Table 3: Multi-seed evaluation of EvoCog on Qwen2.5-7B-Instruct. SR denotes success rate (%), and token counts are averaged per task.

BenchmarkSeed42 SRSeed 43 SRSeed 44 SRSR (Mean ± Std)Output Tokens (Mean ± Std)
ALFWorld92.593.59292.67 ± 0.76562.68 ± 34.41
ScienceWorld76.573.075.575.00 ± 1.80981.42 ± 53.58
WebShop61.462.863.262.47 ± 0.95876.49 ± 47.09

These results demonstrate that EvoCog maintains strong performance across different random seeds, supporting the robustness of our findings.

AuthorsOpenReview ↗

Response for weakness 2

We agree that several details should be made more explicit:

1. Teacher Model.

GPT-4o is used to generate program-conditioned SFT reasoning traces while preserving expert actions, as detailed in Appendix A.2.1.

2. Failure-Sensitive Criteria: SCPO identifies counterfactual anchor states using three criteria.

  • Divergence: The first action divergence between successful and failed trajectories sharing a common state prefix.
  • Recovery: Invalid actions, explicit failure feedback, or nonterminal transitions without observable progress.
  • Stagnation: Repeated state–action pairs, no progress for three consecutive steps.

3. External Diagnosis and Repair Modules.

Both modules are implemented through task-specific GPT-4o prompts without additional training. The diagnosis module produces structured failure classifications, while the repair module generates guard–modification pairs. We will provide the complete prompt templates and output schemas in the revised manuscript.

4. Data Separation.

For ALFWorld and ScienceWorld, we follow the same dataset splits and evaluation protocol as CogRouter[1] to ensure a fair comparison. Both SFT and SCPO use only training data, with no overlap with the test set. Furthermore, SFT and SCPO training tasks are kept disjoint, as detailed in Appendix A.2.1.

5. Implementation Details.

The SCPO training procedure, including rollout collection, counterfactual evaluation, optimization objectives, and pseudocode, is detailed in Appendix A.2.3 and Algorithm 1. Training hyperparameters are provided in Appendix A.3.We will further clarify the remaining implementation details in the revised manuscript.

[1] Think Fast and Slow: Step-Level Cognitive Depth Adaptation for LLM Agents.(ICML 2026)

AuthorsOpenReview ↗

Response for question 1-4

Q #1. Figure 1 and computational cost

Figure 1(b) uses program depth (number of cognitive operations) as a proxy for computational cost. It shows that programs with the same depth can yield substantially different utilities, indicating that performance depends on cognitive composition rather than operation count alone. For example, programs containing different combinations of grounding, planning, and checking may have identical operation counts but yield different task outcomes.

Figure 1(a) further shows that different states favor different programs. For example, states with uncertain object locations benefit from grounding-oriented cognition, while failed transitions favor verification-oriented cognition.

Together, these observations motivate adapting the structure of reasoning rather than selecting a fixed reasoning depth.

Q #2. Executable programs versus fixed reasoning patterns

We agree that each individual program defines a fixed sequence of cognitive operations. However, this does not imply a fixed reasoning pattern for the agent. EvoCog dynamically selects different programs at each decision step based on the current state. Unlike predefined holistic cognitive modes, these programs are composed of reusable atomic operations, allowing different cognitive procedures even at the same reasoning depth.

Q #3. Composition of the SCPO objective

The SCPO objective consists of three components:

  • Executor loss (Lexec\mathcal{L}_{exec}): A token-level PPO clipped loss over valid Executor tokens, with masked averaging.
  • Router loss (Lrouter\mathcal{L}_{router}): A PPO clipped loss for program selection over grammar-valid candidates.
  • Counterfactual loss (Lcf\mathcal{L}_{cf}): A utility-weighted pairwise preference loss comparing incumbent and challenger programs at matched anchor states.

Both PPO objectives use group-normalized advantages derived from cost-aware trajectory utilities. The coefficients α\alpha and λcf\lambda_{cf} control the relative contributions of Router optimization and counterfactual credit. Entropy and KL regularization are further detailed in Appendix A.2.3. We will clarify these components in the revised manuscript.

Q #4. Definition of P1–P10 in Figure 3

P1–P10 denote the ten initial grammar-valid cognitive programs defined in Section 3.2 and enumerated in Appendix A.2.2 (Table 4). Figure 3 groups them into five families: Direct (P1), Grounding (P2), Planning (P3–P4), Plan Verification (P5–P6), and Recovery (P7–P10). We will explicitly include this mapping in the Figure 3 caption and clarify the program definitions in the main text.

Thank you for the detailed response and for running the additional experiments. The rebuttal partially addressed my concerns, and I am raising my score from 2 to 3. The comments below are suggestions for future revisions. No further response is needed, and my updated score will remain unchanged.

  1. Weakness 1. The additional experiments strengthen the evidence for the method’s overall effectiveness, but they do not directly address whether the observed gains come from the proposed algorithmic design. After reading the authors’ responses to the other reviewers and checking the relevant appendix sections, I found evidence that the method’s effectiveness is not tied to a particular base model. That is useful, but it does not isolate the contribution of the proposed design choices. I encourage the authors to include more targeted ablations in a future revision.

  2. Reproducibility. I encourage the authors to move the implementation and experimental details essential for reproducing the results into the main text, so that readers do not need to consult the appendix for that information.

Reviewer_nX4ROpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a computational cognitive space consisting of four meta-operations: GROUND, PLAN, CHECK(), ACT. In this setting, they introduce a new learning method, Sparse Counterfactual Program Optimization (SCPO), for optimizing in this cognitive decision space. Across several datasets (ALFWorld, ScienceWorld, WebShop), a post-trained Qwen2.5-7B model achieves higher success rates than either a larger model (GPT5) or optimization scheme (GRPO). Finally, they introduce test-time refinement of the program to save “repairs” into a program library that can be reused in future actions.

The results are convincing: SCPO seems to be appropriate for planning, where the possible meta-programs are relatively limited. The test-time cognitive evolution of adding to the library is reasonable, but not a core contribution of this work. However, the main constraint is situating a task in terms of cognitive programs: this space is relatively limited, and perhaps only useful for planning with smaller models.

Critical strengths

The SCPO objective is creative and solves the problem of identifying a “good” nearby trajectory that can provide a useful signal after it is rolled out.

The empirical results, analysis, and ablations are clear in explaining why the SCPO objective is helpful for learning planning-type tasks. It is also nice that the token-aware objective can co-exist (and actually contributes to) the improved performance.

Critical weaknesses

The method is only applied over one model (Qwen2.5-7B), and so it is not clear whether SCPO can generalize to larger models or even other model families.

One of the core assumptions is that these four reusable operations and causal grammar is the correct abstraction, but this assumption is neither revisited nor analyzed later.

In this method, there is still a limitation of warming up the LLM with SFT using expert trajectories.

Questions for authors

Why is the Router the same as the Executor? What are the benefits of doing this?

What is the importance of sparse anchor selection in SCPO vs. random sampling of anchor selection (for this task)? This seems to be a fundamental part of SCPO and why SCPO may (or may not) be generalizable to other tasks.

3 comments
AuthorsOpenReview ↗

Response for weakness 1 & 2 & 3

We thank the reviewer for the positive assessment of SCPO and the constructive suggestions. We address the concerns regarding generalizability, design choices, and sparse anchor selection below.


W #1. Generalizability Across Model Families

We clarify that EvoCog has also been evaluated on Llama3.1-8B-Instruct, with results already reported in Appendix A.4 (Table 7). SCPO achieves an average success rate of 78.67%, outperforming CogRouter (74.17%), GRPO (61.27%), and GiGPO (71.40%). Due to computational resource constraints, we have not extended our experiments to substantially larger models, which we leave for future work.

W #2. Design of Atomic Operations and Causal Grammar

The four operations (GROUND, PLAN, CHECK, and ACT) capture reusable cognitive functions reflected in existing methods, such as planning in Plan-and-Act, failure reflection in Reflexion, and state–goal alignment in ReflAct. EvoCog unifies these functions into composable programs, while the causal grammar ensures valid operation dependencies. Figure 1 shows that different compositions yield different utilities even at the same depth, and Figure 3 demonstrates adaptive program usage across environments. We acknowledge that this abstraction is not necessarily optimal, and exploring alternative operation sets and grammars remains future work.

W #3. Dependence on SFT Initialization

We acknowledge that EvoCog relies on expert trajectories for SFT initialization. To investigate this dependency, we additionally train SCPO directly without SFT. As shown in Table 1, removing SFT substantially reduces success rates from 92.5% to 25.5% on ALFWorld, 76.5% to 6.0% on ScienceWorld, and 61.4% to 5.6% on WebShop. This highlights the importance of SFT in establishing reliable program execution before RL-based optimization. Reducing reliance on expert demonstrations remains an important direction for future work.

Table 1. Ablation Study on the Impact of SFT Initialization (Success Rate, %).

MethodALFWorld ↑ScienceWorld ↑WebShop ↑
SCPO w/o SFT25.56.05.6
SCPO (Ours)92.576.561.4
AuthorsOpenReview ↗

Response to Qestion1 & 2

Q #1. Why Share the Router and Executor?

The Router and Executor share the same backbone but perform different functions: the Router selects a cognitive program, while the Executor generates actions conditioned on that program. Sharing the backbone allows both components to leverage the same state representations and avoids maintaining a separate routing model, reducing deployment complexity and model memory overhead.

Q #2. Importance of Sparse Anchor Selection To evaluate the importance of failure-sensitive anchor selection, we compare it with random anchor sampling under the same counterfactual rollout budget. As shown in Table 1, our strategy consistently outperforms random sampling on ALFWorld (+4.5 pp), ScienceWorld (+2.0 pp), and WebShop (+5.8 pp). By prioritizing failure-sensitive and critical decision states, our approach provides more informative counterfactual signals than random sampling. The consistent improvements across three distinct environments further support the applicability of sparse anchor selection beyond a single task.

Table 1. Ablation Study of Anchor Selection Strategies (Success Rate, %).

Anchor StrategyALFWorld ↑ScienceWorld ↑WebShop ↑
Random Anchor Selection88.074.555.6
Failure-sensitive Anchor Selection (Ours)92.576.561.4
Improvement (pp)+4.5+2.0+5.8

Thanks, this addresses my concerns around soundness. W2 and W3 are still limitations, in my opinion, of the methods and findings themselves rather than soundness of experiments. After reading the other reviews, I maintain my positive score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.