acceptodds
Desk rejected at ICLR 2027

Self-Evaluating Recursive Agents

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Recursive language-model agents decompose tasks and delegate subtasks to child instances of the same policy, forming a tree of work. Training them, however, is hard: the final outcome is verifiable, but the self-invented intermediate subtasks are numerous and carry no ground truth. Existing methods score each node with a verifier or judge, which is costly at scale and blind to decomposition quality. We argue that a recursive agent must learn three coupled capabilities within one set of weights: decomposing problems into subtasks, solving them, and evaluating the outcomes, each requiring its own training signal. SERA (Self-Evaluating Recursive Agents) turns evaluation into a learned capability of the policy itself. Before delegating, the parent writes a rubric of weighted success criteria for each child subtask; a ranking objective against verified outcomes then trains rubric generation so that the criteria track genuine subtask success. In addition, a complementary leaf-coverage signal provides direct credit for task decomposition. Our central finding is that training the policy to generate aligned rubrics is what drives the gains: because the same weights both evaluate and execute, learning to judge subtasks sharpens the agent's ability to solve them. Notably, external supervision is also reduced: the judge is consulted only to train the rubric generator, while solving is trained against the agent's own rubric scores, which outperform direct use of the judge. Beyond training, the learned rubric doubles as an inference-time selector for tree search. On TextCraft-Synth and TextWorld-Sync, SERA improves over strong recursive-agent baselines by and points on average, and rubric-guided tree search at inference adds a further points on TextWorld-Sync.

Primary area:
reinforcement learning
Keywords:
reinforcement learning, on-policy RL, multi-agent, LLM

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_F4XMOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

SERA trains a recursive-agent policy to execute subtasks, generate evaluation rubrics, and choose decompositions through alternating reinforcement-learning objectives. Rubric generation is by ranking counterfactual subtask rollouts against verified outcomes, Two benchmarks show improved task success, fewer external supervision calls.

Critical strengths
  1. The method addresses a concrete supervision problem: self-generated subtasks lack standard intermediate labels. Committing rubrics before execution and grounding their training in verified within-subtask rankings provides a coherent way to learn reusable local feedback (Sections 3.2–3.4).

  2. The gains are supported by three-seed evaluation and component ablations. SERA improves over RAO by 5.38 and 13.14 percentage points on the two benchmarks, and reduces external judge calls from 147.80K to 35.79K on TextWorld-Sync (Tables 1–2).

Critical weaknesses
  1. All policy-training experiments use only Qwen3-4B-Instruct-2507 (Section 4). Without a second base model, it is unclear whether the gains are robust across models or depend on this specific policy. Validation on another model, such as a comparably sized model from a different family, would strengthen the evidence for generality.

  2. The evaluation is confined to TextCraft-Synth and TextWorld-Sync, both synthetic text environments. These support controlled analysis, but do not establish effectiveness on realistic terminal or repository-level tasks. Results on a representative benchmark, such as Terminal-Bench 2.0 or SWE-Bench Pro, would strengthen the evidence for broader agent applicability. .

Questions for authors
  1. Why all experiments conducted only on Qwen3-4B-Instruct-2507? Do the gains also hold for another base model?

  2. beyond TextCraft-Synth and TextWorld-Sync,whether the method benefit for Terminal-Bench 2.0 or SWE-Bench Pro?

3 comments

W1: Generalization Across Different Model Backbones

Reviewer Concern: Why are all experiments conducted only on Qwen3-4B-Instruct-2507? Do the improvements also hold for another backbone?

Response: Thank you for this valuable suggestion. To assess the generalizability of SERA across model backbones, we conducted additional experiments using Qwen3.5-4B as a second backbone on the TextCraft benchmark. We evaluated each method over three independent runs to account for randomness and report the mean ± standard deviation.

MethodEasy ↑Medium ↑Hard ↑Extreme ↑Avg ↑
Qwen3.5-4B75.28 ± 1.4212.99 ± 0.980.00 ± 0.000.00 ± 0.0021.89 ± 0.66
GRPO94.78 ± 1.0419.41 ± 0.540.00 ± 0.000.00 ± 0.0028.55 ± 0.37
RAO98.64 ± 1.1896.09 ± 1.3692.89 ± 1.8510.29 ± 1.4777.53 ± 1.10
SERA (Ours)98.87 ± 1.0499.53 ± 0.4797.30 ± 1.7026.96 ± 2.5883.28 ± 0.24

Key findings. SERA consistently outperforms RAO across all four difficulty levels on Qwen3.5-4B. Specifically:

  • Overall improvement: SERA achieves an average success rate of 83.28%, outperforming RAO (77.53%) by +5.75 percentage points.
  • Substantial gains on challenging tasks: On the Extreme subset, SERA improves the success rate from 10.29% to 26.96%, a +16.67 percentage-point gain.
  • Consistent performance: The relatively small standard deviations across three independent runs indicate that the improvements are stable.

W2: Generalization Beyond TextCraft-Synth and TextWorld-Sync

Reviewer Concern: Does SERA also provide benefits on more realistic agentic benchmarks, such as Terminal-Bench 2.0 or SWE-Bench Pro?

Response: Thank you for raising this important question. To investigate whether SERA generalizes beyond TextCraft-Synth and TextWorld-Sync, we conducted an additional evaluation on Terminal-Bench 2.1, a benchmark featuring realistic terminal-based engineering tasks.

Experimental setup. We use DeepSeek-V4.1-Flash-1017 as the base model, using the mini-swe-agent harness with the maximum 256K context. We incorporate SERA's core mechanism—recursive subtask delegation with self-evaluation—into the agent's execution workflow: At each step, the parent policy decides whether to solve the task directly or delegate a focused subtask to a child instance, and then self-evaluates the child’s returned work against the subtask criteria to either accept the result or trigger a retry.

On Terminal-Bench 2.1, adding SERA-style recursive delegation and self-evaluation raises the pass rate from 56.2% to 67.4% (+11.2 points). This shows that in-policy evaluation transfers effectively to realistic terminal engineering tasks and provides a substantial gain through test-time verification and retry.

Supplement to W1: Generalization Across Different Model Backbones

Reviewer Concern: Why are all experiments conducted only on Qwen3-4B-Instruct-2507? Do the improvements also hold for another backbone?

Response: Following our earlier response, we conducted additional full RL experiments using Granite-4.2-8B, a larger backbone from a different model family, on TextWorld-Sync, to further validate SERA's cross-family generalizability.

MethodEasy ↑Medium ↑Hard ↑Extreme ↑Avg. ↑
Granite-4.2-8B64.67 ± 0.9219.43 ± 1.310.00 ± 0.000.00 ± 0.0021.02 ± 0.51
GRPO99.71 ± 0.2965.81 ± 2.2912.95 ± 0.870.00 ± 0.0044.62 ± 0.86
RAO99.43 ± 0.2995.43 ± 1.5938.86 ± 1.8710.29 ± 1.3161.00 ± 0.84
SERA (Ours)100.00 ± 0.0099.33 ± 0.3363.81 ± 2.7224.95 ± 1.1972.02 ± 0.39

Key findings. SERA's full RL training benefits generalize to a larger, non-Qwen backbone.

  • Overall improvement: SERA achieves 72.02% average success, outperforming RAO (61.00%) by +11.02 percentage points.
  • Challenging tasks: SERA improves Hard and Extreme success by +24.95 and +14.66 percentage points, respectively.

Your Feedback Would Be Appreciated

Dear Reviewer F4XM,

As the discussion period closes in approximately five hours, we would like to kindly follow up on your concerns.

We have conducted additional experiments on Granite-4.2-8B (+11.02 points over RAO) and Terminal-Bench 2.1 (+11.2 points), directly addressing model and task generalizability.

We hope these results resolve your concerns and would sincerely appreciate your reconsideration of our work.

Thank you for your thoughtful feedback.

Best regards,

Authors

Reviewer_gXEtOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work aims to enhance the capabilities of recursive agents in handling complex tasks. To address the limitations of prior approaches, which often rely on external evaluators that incur substantial costs and provide limited granularity in assessment, the authors propose training a model that can decompose tasks, execute subtasks, and perform fine-grained self-evaluation. This self-evaluation capability is integrated into both the training and inference stages.

The paper's primary contribution is the generation of a dedicated scoring rubric for each subtask. The authors investigate this component extensively through experiments, demonstrating its effectiveness. Overall, this is a solid piece of work with a well-motivated approach. However, as the experimental setup primarily focuses on validating the effectiveness of the proposed method, further analysis would strengthen the paper. In particular, a more thorough investigation of potential failure modes and a deeper analysis of the underlying mechanisms would provide greater insight into why and when the method works.

Critical strengths

The proposed approach is well motivated. Enabling the model to explicitly define the criteria for an excellent subtask outcome can help improve both execution quality and task decomposition. The results demonstrate that SERA performs well, reducing reliance on costly external verifiers while improving overall agent capabilities.

Critical weaknesses

The experiments evaluate the proposed method using a single model across two datasets that appear to share substantial similarities. Consequently, it remains unclear whether the observed benefits generalize to a broader range of benchmarks, model architectures, and real-world tasks. In addition, the paper does not appear to provide a direct evaluation of rubric quality, particularly the accuracy of rubric segmentation and the reliability of the resulting scores. Without such an analysis, it is difficult to determine how much the rubrics contribute to the overall performance gains and whether their benefits justify the additional computational costs.

Questions for authors
  1. How does SERA perform on a broader range of benchmarks? The authors need to evaluate its generalizability across different backbone models and more diverse real-world tasks, such as code generation and web search.

  2. What additional computational overhead does SERA introduce during inference? A breakdown of the costs associated with rubric generation, execution-trace analysis, item-level scoring, and other relevant operations would help clarify the method's efficiency. It would also be useful to quantify the computational costs of counterfactual sampling during training.

  3. How accurately does SERA segment subtasks and generate the corresponding scoring rubrics? How reliable are the scores assigned by these rubrics compared with those produced by human evaluators or external verifiers? Such an evaluation would help isolate the contribution of rubric quality to the overall performance improvements.

8 comments

[1/3] W1: Generalization Across Model Backbones and Diverse Tasks

Reviewer Concern: The experiments use a single model on two similar datasets. How does SERA generalize across different backbone models and more diverse real-world tasks, such as code generation and web search?

Response: Thank you for this insightful comment. To further evaluate the generalizability and robustness of SERA, we conducted additional reinforcement learning (RL) experiments using Qwen3.5-4B as a second backbone and extended our evaluation to DeepSearch, a substantially different task domain involving knowledge retrieval and multi-hop reasoning. All results are reported as the mean ± standard deviation over three independent evaluation runs.

(1) Generalization Across Model Backbones

To investigate whether SERA's improvements extend beyond the original Qwen3-4B backbone, we trained and evaluated Qwen3.5-4B on TextCraft under the same experimental setting.

MethodEasy ↑Medium ↑Hard ↑Extreme ↑Avg. ↑
Qwen3.5-4B75.28 ± 1.4212.99 ± 0.980.00 ± 0.000.00 ± 0.0021.89 ± 0.66
GRPO94.78 ± 1.0419.41 ± 0.540.00 ± 0.000.00 ± 0.0028.55 ± 0.37
RAO98.64 ± 1.1896.09 ± 1.3692.89 ± 1.8510.29 ± 1.4777.53 ± 1.10
SERA (Ours)98.87 ± 1.0499.53 ± 0.4797.30 ± 1.7026.96 ± 2.5883.28 ± 0.24

Key findings:

  • Consistent improvements across difficulty levels. SERA outperforms RAO on all four difficulty subsets, achieving an overall success rate of 83.28%, compared with 77.53% for RAO (+5.75 percentage points).
  • Substantial gains on challenging tasks. On the Extreme subset, SERA improves the success rate from 10.29% to 26.96%, yielding a +16.67 percentage-point improvement.

(2) Generalization to DeepSearch: Knowledge Retrieval and Multi-Hop Reasoning

To further examine SERA's applicability beyond the original task environments, we conducted additional RL experiments with Qwen3.5-4B in the DeepSearch dataset.

MethodNQ ↑TriviaQA ↑PopQA ↑HotpotQA ↑2Wiki ↑MuSiQue ↑Bamboogle ↑Avg. ↑
Qwen3.5-4B31.75 ± 1.0654.00 ± 1.4137.25 ± 3.1826.75 ± 1.0621.00 ± 0.007.25 ± 1.7725.60 ± 2.2629.09 ± 0.89
GRPO37.00 ± 2.1262.00 ± 2.8342.75 ± 1.0642.00 ± 2.8340.50 ± 3.5413.25 ± 0.3545.20 ± 0.5740.39 ± 0.38
RAO36.17 ± 1.6160.67 ± 2.5242.33 ± 1.6142.33 ± 1.5350.83 ± 1.6119.67 ± 2.5745.60 ± 2.1242.51 ± 0.63
SERA (Ours)37.17 ± 1.8968.00 ± 0.5044.17 ± 2.3646.00 ± 1.5052.33 ± 2.0218.83 ± 1.2649.87 ± 0.9245.20 ± 0.26

Key findings:

  • Consistent gains across diverse QA benchmarks. SERA achieves the highest performance on six out of seven benchmarks, covering both factual retrieval and multi-hop reasoning tasks.
  • Improved overall performance. SERA achieves an average success rate of 45.20%, outperforming RAO (42.51%) by +2.69 percentage points and GRPO (40.39%) by +4.81 percentage points.

[2/3] W2: Inference-Time Computational Overhead of SERA

Reviewer Concern: What additional computational overhead does SERA introduce during inference? A breakdown of rubric generation, execution-trace analysis, item-level scoring, and other operations would help clarify its efficiency.

Response: Thank you for this valuable suggestion. We would like to clarify that SERA introduces no additional overhead from rubric generation, execution-trace analysis, item-level scoring, or counterfactual forking during standard inference. These operations are performed exclusively during training, and their computational costs have already been analyzed in Table 1 of the original submission.

To provide a more comprehensive understanding of SERA's inference-time efficiency, we additionally analyze its token consumption and subagent execution patterns on TextWorld. Specifically, we report the average total inference tokens (input + output, in thousands) across difficulty levels, along with the average number of subagent calls and execution steps per subagent.

MethodEasy ↓Medium ↓Hard ↓Extreme ↓Avg. Tokens (K) ↓Subagent Calls ↓Steps per Subagent
RAO59.99255.89382.12359.84264.467.829.13
SERA (Ours)110.92312.11435.06436.46323.645.6713.43

Key findings:

  • Fewer but longer subagent executions. Compared with RAO, SERA reduces the average number of subagent calls from 7.82 to 5.67 (−27.5%), while increasing the average number of execution steps per subagent from 9.13 to 13.43 (+47.1%). This suggests a shift toward fewer, more sustained subagent executions.

  • Moderately higher token consumption. SERA consumes 323.64K tokens per task, compared with 264.46K for RAO, representing a 22.4% increase. This additional cost is associated with longer subagent execution trajectories rather than the training-specific evaluation procedures.

In summary, SERA does not introduce additional inference-time costs from its training-specific evaluation mechanisms. However, it incurs a 22.4% increase in total inference tokens relative to RAO, alongside a 27.5% reduction in subagent calls. These results highlight a trade-off between token consumption and subagent execution patterns: SERA tends to invoke subagents less frequently while allowing each invocation to proceed for more steps.

[3/3] W3: Rubric Quality and Score Reliability

Reviewer Concern: How reliable are the generated rubrics and their scores? Do the improvements justify the additional computational costs?

Response: Thank you for this valuable suggestion. We conduct a controlled comparison of rubric quality and scoring reliability across 500 matched TextCraft subagent delegation contexts. Given identical delegation contexts, Qwen3-4B-Instruct-2507 and SERA each generate a rubric for the delegated subtask, then use their respective rubrics to score the same subagent execution trajectories.

We evaluate six dimensions from two complementary perspectives:

(1) Scoring Reliability. We measure the Mean Score Gap between verifier-successful and verifier-failed trajectories within each delegation. A larger gap indicates stronger average score separation and higher reliability.

(2) Rubric Content Quality. We use rule-assisted checks and targeted semantic review to evaluate:

  • Clarity: Whether each item expresses a single coherent judgment without coupling independent requirements.
  • Verifiability: Whether criteria can be assessed using observable evidence.
  • Meaningfulness: Whether criteria assess task-relevant execution quality.
  • Correctness: Whether requirements comply with task facts and environment rules.
  • Richness: Diversity and balance of evaluation criteria across eight themes, measured using the entropy-based effective number of themes.

The first four content metrics use 0/5/10 item-level scores, which are first averaged within each rubric and then equally across rubrics. Richness is computed separately from the overall thematic distribution, counting each theme at most once per rubric. Both content scoring and thematic classification are performed by GPT-6 Astra.

Metric ↑Qwen3-4B-InstructSERAΔ\Delta
Reliability0.17760.4263+0.2487
Clarity6.807.93+1.13
Verifiability7.447.49+0.05
Meaningfulness7.767.89+0.13
Correctness7.657.56-0.09
Richness6.8629.070+2.208

Key findings. SERA demonstrates stronger score discrimination alongside clearer and more diverse evaluation criteria. The Mean Score Gap increases from 0.1776 to 0.4263 (+0.2487), potentially benefiting from improvements in Clarity (+1.13) and Richness (+2.208). These results suggest that more coherent criteria and broader evaluation coverage may help distinguish successful from failed executions, while the other three content metrics remain largely unchanged.

Reviewer_ue5LOpenReview ↗
Rating
4(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work closely follows Recursive Agent Optimization (RAO) [1], an algorithm for training recursive agents/RLMs with reinforcement learning and denser credit assignment. The main variations of this paper on top of the RAO experiments are: (1) Reward design heuristics to encourage better task decomposition; (2) Training the model to self-verify sub-agent outcomes by generating rubrics for sub-tasks; (3) Alternating optimization during across different advantage types.

[1] Gandhi et al., Recursive Agent Optimization, COLM 2026.

Critical strengths
  • Multi-agent training and bootstrapping via RL and better reward design is a highly relevant topic for agent-inference time scaling.
  • The experiments show improvements with the variations over the original settings in the RAO paper.
  • The breakdown of rewards and advantages into execution, decomposition and rubric generation is interesting.
Critical weaknesses
  • The benchmarks used are synthetic and designed to be clean and simple. It is not clear if the design choices (e.g., self-verification) would work as well on more complex scenarios (e.g., deep research, science/swe agents, etc.), and especially so with the small model size experimented with.
  • The method still relies on an external llm judge or verifier to get signal to train self-verification. This relies on either noisy LLM judges or requires deterministic verifiers for sub-tasks don't exist in the general case since sub-tasks are generated on the fly.
  • Sampling M subtrees per sub-task for rubric training is computationally expensive, especially on more real-world tasks. But despite paying this cost, interestingly the authors do not use this to compute the baseline for that node's execution advantage.
  • The decomposition reward/advantage does not seems to hurt on TextCraft hard and Extreme tasks where decomposition is pretty natural. This is a bit puzzling. (Table 1)
  • The execution advantage (eq 4) is identical to that of RAO, but the paper does not attribute/mention this.
Questions for authors
  • Why not use the M sampled subtrees to compute a more calibrated baseline for sub-trajectory advantages?
6 comments

[1/5] W1. Benchmark Complexity and Generalization to Real-World Agentic Tasks

Reviewer Concern: The benchmarks used are synthetic and designed to be clean and simple. It is not clear whether the proposed design choices (e.g., self-verification) would work equally well in more complex scenarios, such as deep research, scientific reasoning, or software engineering agents, especially given the relatively small model size.

Response: Thank you for this insightful comment. We agree that evaluating SERA beyond synthetic environments is important for establishing its broader applicability. To address this concern, we conducted two additional sets of experiments covering (1) knowledge-intensive retrieval and multi-hop reasoning and (2) realistic terminal-based software engineering tasks.

(1) Generalization to DeepSearch: Knowledge Retrieval and Multi-Hop Reasoning

We first evaluated SERA on DeepSearch using Qwen3.5-4B as the backbone. Unlike the original synthetic environments, these tasks require knowledge retrieval, multi-step reasoning, and integration of information from multiple sources.

We conducted additional reinforcement learning (RL) experiments and evaluated each method over three independent runs, reporting the mean ± standard deviation.

MethodNQ ↑TriviaQA ↑PopQA ↑HotpotQA ↑2Wiki ↑MuSiQue ↑Bamboogle ↑Avg. ↑
Qwen3.5-4B31.75 ± 1.0654.00 ± 1.4137.25 ± 3.1826.75 ± 1.0621.00 ± 0.007.25 ± 1.7725.60 ± 2.2629.09 ± 0.89
GRPO37.00 ± 2.1262.00 ± 2.8342.75 ± 1.0642.00 ± 2.8340.50 ± 3.5413.25 ± 0.3545.20 ± 0.5740.39 ± 0.38
RAO36.17 ± 1.6160.67 ± 2.5242.33 ± 1.6142.33 ± 1.5350.83 ± 1.6119.67 ± 2.5745.60 ± 2.1242.51 ± 0.63
SERA (Ours)37.17 ± 1.8968.00 ± 0.5044.17 ± 2.3646.00 ± 1.5052.33 ± 2.0218.83 ± 1.2649.87 ± 0.9245.20 ± 0.26

Key findings:

  • Consistent improvements across diverse QA tasks. SERA achieves the best performance on six out of seven benchmarks, covering both factual knowledge retrieval and multi-hop reasoning.
  • Higher overall success rate. SERA achieves an average success rate of 45.20%, outperforming RAO (42.51%) by +2.69 percentage points and GRPO (40.39%) by +4.81 percentage points.
  • Effectiveness with a compact backbone. These improvements are achieved using Qwen3.5-4B, providing additional evidence that SERA remains effective in more complex reasoning environments without requiring a substantially larger model.

(2) Generalization to Terminal-Bench 2.1: Realistic Software Engineering Tasks

To further investigate the applicability of SERA's core mechanism in realistic agentic environments, we conducted an additional evaluation on Terminal-Bench 2.1, which involves complex terminal-based engineering tasks requiring multi-step execution, debugging, and interaction with external tools.

Experimental setup. We use DeepSeek-V4.1-Flash-1017 as the base model, integrated with the mini-swe-agent harness and a maximum context length of 256K tokens. We incorporate SERA's core mechanism—recursive subtask delegation with self-evaluation—into the agent's execution workflow:

  1. Adaptive delegation: At each decision point, the parent agent determines whether to solve the task directly or delegate a focused subtask to a child agent.
  2. In-policy self-evaluation: The parent agent evaluates the child's returned work against the specified subtask requirements.
  3. Evaluation-guided retry: If the returned result does not satisfy the requirements, the parent agent can initiate a retry rather than immediately accepting the result.

Results. Incorporating SERA-style recursive delegation and self-evaluation increases the pass rate on Terminal-Bench 2.1 from 56.2% to 67.4%, yielding an absolute improvement of +11.2 percentage points.

This substantial improvement provides further evidence that SERA's core delegation and self-evaluation mechanisms can benefit realistic software engineering workflows.

[2/5] W2. Reliance on External Verifiers or Noisy LLM Judges for Subtask Rubric Training

Concern. The method still relies on an external LLM judge or verifier to get signal to train self-verification. This relies on either noisy LLM judges or requires deterministic verifiers for sub-tasks that don't exist in the general case since sub-tasks are generated on the fly.

Response. We fully agree that deterministic verifiers rarely exist for on-the-fly subtasks in general domains. That is precisely why we paired TextCraft-Synth with TextWorld-Sync, where no subtask verifiers exist and intermediate supervision comes solely from an external LLM judge—showing that SERA outperforms direct LLM-judge rewards (RAO) by +13.14 points while cutting LLM-judge calls by 4.1$\times$.

  1. Testing the verifier-free subtask regime (TextWorld-Sync): As noted in Section 4 (lines 386–390) and Appendix C.2 (lines 991–993), arbitrary policy-invented subtasks in TextWorld-Sync cannot be verified by the root evaluator. Both RAO and SERA therefore rely on the exact same external LLM judge for subtask labels.
  2. Why SERA is far more robust to noisy LLM judges than direct judge supervision (RAO): On TextWorld-Sync, SERA achieves 65.07% (SERA w/o D: 60.26%) versus 51.93% for RAO (Table 1). Channeling the noisy LLM judge into rubric training outperforms using the judge directly as the execution reward for two structural reasons:
    • Ordinal pairwise filtering vs. raw cardinal rewards (Eq. 3): RAO injects noisy binary judge labels directly into the execution policy gradient at every node. In contrast, SERA uses the LLM judge only in Eq. (3) to form ordered pairs P={(i,j):yi>yj}P = \{(i, j) : y_i > y_j\} among BB counterfactual rollouts of the exact same subtask started from the same cloned state. Even if an LLM judge is noisy or poorly calibrated across different subtasks, its relative ordering between contrasting attempts on the identical subtask provides a reliable ordinal anchor, while the solver trains on smooth, dense rubric scores sv∈[0,1]s_v \in [0, 1] gated by subtask completion gv∈{0,1}g_v \in \{0, 1\}.
    • Representation transfer across shared weights: Comparing RAO + RT (45.62%) against SERA w/o D (60.26%) in Table 1 shows that training the policy to formulate aligned evaluation criteria and using them as execution rewards directly improves how the shared weights solve subtasks.
  3. Why sparse external anchoring is both necessary and lightweight: Removing external anchoring completely (SERA w/o D & RT: 51.10%, Table 1) or replacing verified ranking with label-free score dispersion (Rank+Disc.: 65.77% vs. Rank: 74.84%, a −9.07-9.07-point drop in Table 3) degrades performance because shared generator/solver weights can game unanchored criteria. By invoking the external judge on only 2 out of every 20 updates (10%), SERA reduces LLM judge calls from 147.80K to 35.79K (4.1$\times$) and input tokens from 633.80M to 125.32M (5.1$\times$) during training (Table 2), and requires zero external judge calls at inference time (Table 4).

Revision. We will revise Section 3.4 and Section 4.1 to emphasize how TextWorld-Sync evaluates the verifier-free subtask setting and explain why ordinal counterfactual ranking (Eq. 3) acts as a structural denoiser for imperfect LLM judges.

[3/5] W3. Why Counterfactual Subtask Rollouts Are Not Used for the Execution Advantage Baseline (Eq. 4) & Computational Cost

Concern. Sampling M subtrees per sub-task for rubric training is computationally expensive, especially on more real-world tasks. But despite paying this cost, interestingly the authors do not use this to compute the baseline for that node's execution advantage.

Response. We apologize that our presentation in Sections 3.4–3.5 left this ambiguous: under our alternating schedule (16 : 2 : 2, Section 3.6 & Table 6), counterfactual subtask rollouts (B=8B=8 per delegation point) are sampled only during the 10% Rubric-Training (RT) updates, and are never sampled during the 80% Execution updates where Ai,vexecA^{\text{exec}} _ {i,v} (Eq. 4) is computed.

  1. Why counterfactual rollouts do not exist during Execution updates: As detailed in Section 3.4 (lines 282–284: "this cloning and counterfactual sampling is required only for rubric-generation training; the execution and decomposition phases operate on ordinary rollouts without additional sampling") and Appendix B.2 (Table 6), each optimizer step activates only one stage:
    • During the 16 Execution steps (80% of updates) and 2 Decomposition steps (10% of updates), the policy generates ordinary recursive trees without state cloning (B=1B=1 rollout per delegated subtask). Thus, when Ai,vexecA^{\text{exec}} _ {i,v} (Eq. 4) is computed and backpropagated, sibling counterfactual rollouts of node vv do not exist, which is why Eq. (4) uses the cross-tree LOO root outcome baseline 1K−1∑j≠iRj\frac{1}{K-1}\sum _ {j\neq i} R _ j.
    • State cloning and sampling B=8B=8 counterfactual rollouts (where BB is the per-subtask rollout group size in Section 3.4 / Table 5, and MM in Eq. 6 is the number of valid delegation events in the batch) occur exclusively during the 2 RT steps (10% of updates). During RT updates, execution tokens are masked out (Excluded from loss, Figure 1 Stage 3) and only rubric-generation tokens receive gradients via Eq. (6).
  2. Why we do not also run execution updates on the 10% RT steps using the BB rollouts as a local baseline:
    • Gradient separation: Section 3.6 decouples execution updates from rubric updates across steps so that solver and evaluator gradients do not co-adapt on the same batch.
    • Baseline semantics: Centering node vv's execution reward by the mean over BB rollouts of the same delegated subtask XvX _ v would remove the signal of whether the parent's subtask specification XvX _ v was well-posed relative to the root task and would place child nodes on a different baseline scale than the root node.
  3. Actual computational and wall-clock cost (Table 2 & Table 7):
    • Because counterfactual branching occurs on only 10% of updates—and because the B−1B-1 counterfactual rollouts execute only the immediate child subtask without recursively expanding deeper subtrees (Section 3.4, lines 280–282)—the sampling overhead is small.
    • In Appendix B.3 (Table 7, measured on identical 2×82\times 8 H200 GPUs under the same total update budget), SERA (24h 06m) is actually faster in wall-clock time than RAO (29h 36m) on TextCraft-Synth while using 2.1$\times$ fewer verifier calls (Table 2). On TextWorld-Sync, serialized shared-state locks increase wall-clock time from 13h 08m (RAO) to 24h 14m (SERA), while reducing external LLM judge calls by 4.1$\times$ and yielding a +13.14-point accuracy improvement.

Revision. We will add an explicit clarification in Section 3.4 and immediately after Eq. (4) in Section 3.5 stating that the BB counterfactual rollouts are generated only during the 10% RT steps (where execution loss is masked) and not during the 80% Execution steps, along with a forward pointer to the wall-clock runtime comparison in Appendix B.3 (Table 7).

Comments on the paper

4 comments

Dear ACs and Reviewers, We sincerely thank you for the time and effort in evaluating our paper. To facilitate the next stage of discussion, we would like to (1) summarize our paper and our contribution; (2) the key points from the rebuttal as below:

Summary of Our Paper and Contributions

Recursive language-model agents decompose complex problems and delegate subtasks to child instances of the same policy, forming a dynamic tree of work. However, training recursive agents via reinforcement learning remains challenging: while the root outcome is verifiable, policy-invented intermediate subtasks lack ground-truth verifiers, and scoring every node with an external verifier or LLM judge is coarse, costly at scale, and blind to decomposition quality. To address these challenges, we propose SERA (Self-Evaluating Recursive Agents) through: (1) formulating recursive-agent training as learning three coupled capabilities—subtask decomposition, execution, and self-evaluation—within a single shared policy via alternating optimization; (2) introducing an ex-ante subtask-specific rubric mechanism trained via a counterfactual ranking objective against verified outcomes, which turns evaluation into an in-policy capability that sharpens execution while amortizing external supervision; (3) designing a leaf-coverage decomposition reward that directly credits structurally meaningful delegation; and (4) demonstrating through extensive experiments and rubric-guided inference-time tree search that SERA achieves superior task success while substantially reducing reliance on external evaluators.

The key points from the rebuttal

Overall, the reviewers provide positive and constructive comments on our paper.

Key Strengths Highlighted by Reviewers

Reviewer ue5L Multi-agent training and bootstrapping via RL is a highly relevant topic for agent inference-time scaling, the breakdown of rewards and advantages into execution, decomposition, and rubric generation is interesting and clear, and the experiments demonstrate clear improvements over RAO.

Reviewer gXEt The proposed approach is solid and well-motivated, enabling the model to explicitly define subtask criteria improves both execution quality and task decomposition, and SERA reduces reliance on costly external verifiers while integrating self-evaluation into both training and inference stages.

Reviewer F4XM The method addresses a concrete supervision problem where self-generated subtasks lack standard intermediate labels, committing rubrics before execution and grounding training in verified within-subtask rankings provides a coherent way to learn reusable local feedback, and the performance gains and reduced external judge calls are well-supported by multi-seed evaluation and component ablations.

How We Addressed Reviewer Concerns

Reviewer ue5L:

  • Real-world generalization ([1/5] W1): We conducted additional experiments on DeepSearch and Terminal-Bench 2.1, demonstrating improvements on knowledge-intensive reasoning and realistic terminal engineering tasks.
  • External supervision ([2/5] W2): We clarified how SERA learns from imperfect LLM judges while substantially reducing external Judge calls.
  • Counterfactual sampling ([3/5] W3): We explained that additional branches are sampled only during rubric-training updates and provided computational cost analysis.
  • Decomposition training ([4/5] W4): We clarified the effects of decomposition rewards across task difficulties and training distributions.
  • Equation attribution ([5/5] W5): We acknowledged RAO's execution-advantage formulation and added the appropriate citation.

We sincerely appreciate the reviewer's positive follow-up and decision to increase the rating from 3 to 4\textcolor{red}{\text{3 to 4}}.


Reviewer gXEt:

  • Cross-family generalization ([1/3] W1): We conducted additional RL experiments using Qwen3.5-4B on TextCraft and DeepSearch. Following the reviewer's further concern, we performed full RL training using Granite-4.2-8B, a non-Qwen backbone, on TextWorld-Sync. SERA achieves 72.02% average success, outperforming RAO (61.00%) by +11.02 percentage points, providing direct evidence of cross-family generalization.
  • Realistic terminal tasks: We demonstrated the transferability of SERA's core mechanisms to DeepSeek on Terminal-Bench 2.1 (+11.2 points) without additional RL training.
  • Inference efficiency ([2/3] W2): We analyzed inference token consumption and subagent execution patterns, clarifying the trade-off between computational cost and execution behavior.
  • Rubric quality ([3/3] W3): We conducted evaluations demonstrating improved score discrimination (Mean Score Gap: 0.1776 to 0.4263\textcolor{red}{\text{0.1776 to 0.4263}}), rubric clarity, and thematic diversity.

Reviewer F4XM:

  • Backbone generality (W1): We conducted additional RL experiments using Qwen3.5-4B on TextCraft and Granite-4.2-8B on TextWorld\textcolor{red}{\text{Granite-4.2-8B on TextWorld}}. SERA outperforms RAO by +11.02 points\textcolor{red}{\text{+11.02 points}} on Granite-4.2-8B, demonstrating that the full RL training benefits extend beyond the Qwen family.
  • Realistic agent applicability (W2): We evaluated SERA-style recursive delegation and self-evaluation on Terminal-Bench 2.1, improving success from 56.2% to 67.4% (+11.2 points) with DeepSeek.

We sincerely thank the Area Chairs and Reviewers for their constructive feedback and consideration of our additional experiments.

With sincere respect and gratitude,

Authors

Update to Area Chairs: Rebuttal Progress

Dear Area Chairs,

We sincerely appreciate that two reviewers have increased their ratings following our rebuttal:

  • Reviewer ue5L: Increased the rating from 3 to 4.
  • Reviewer gXEt: Increased the rating from 2 to 3, explicitly stating that our additional experiments resolved their concerns.

Overall, the reviewer ratings have improved from (3, 2, 2) to (4, 3, 2), bringing the average rating to 3.00.

Reviewer F4XM's main concerns substantially overlap with those of Reviewer gXEt, and we have provided additional experimental evidence addressing both:

  • Cross-family and backbone generalization: SERA outperforms RAO by +11.02 points on TextWorld-Sync with Granite-4.2-8B (non-Qwen) and +5.75 points on TextCraft with Qwen3.5-4B, demonstrating gains across different backbones.
  • Real-world task generalization: On DeepSearch, SERA improves average success by +2.69 points across seven knowledge-retrieval reasoning benchmarks. On Terminal-Bench 2.1, SERA's core mechanisms improve success by +11.2 points on realistic terminal engineering tasks.

We hope these additional results and positive reviewer feedback will be considered in the final assessment.

Thank you for your time and consideration.

Best regards,

Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.