- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
SERA trains a recursive-agent policy to execute subtasks, generate evaluation rubrics, and choose decompositions through alternating reinforcement-learning objectives. Rubric generation is by ranking counterfactual subtask rollouts against verified outcomes, Two benchmarks show improved task success, fewer external supervision calls.
Critical strengths
-
The method addresses a concrete supervision problem: self-generated subtasks lack standard intermediate labels. Committing rubrics before execution and grounding their training in verified within-subtask rankings provides a coherent way to learn reusable local feedback (Sections 3.2–3.4).
-
The gains are supported by three-seed evaluation and component ablations. SERA improves over RAO by 5.38 and 13.14 percentage points on the two benchmarks, and reduces external judge calls from 147.80K to 35.79K on TextWorld-Sync (Tables 1–2).
Critical weaknesses
-
All policy-training experiments use only Qwen3-4B-Instruct-2507 (Section 4). Without a second base model, it is unclear whether the gains are robust across models or depend on this specific policy. Validation on another model, such as a comparably sized model from a different family, would strengthen the evidence for generality.
-
The evaluation is confined to TextCraft-Synth and TextWorld-Sync, both synthetic text environments. These support controlled analysis, but do not establish effectiveness on realistic terminal or repository-level tasks. Results on a representative benchmark, such as Terminal-Bench 2.0 or SWE-Bench Pro, would strengthen the evidence for broader agent applicability. .
Questions for authors
-
Why all experiments conducted only on Qwen3-4B-Instruct-2507? Do the gains also hold for another base model?
-
beyond TextCraft-Synth and TextWorld-Sync,whether the method benefit for Terminal-Bench 2.0 or SWE-Bench Pro?
3 comments
Supplement to W1: Generalization Across Different Model Backbones
Reviewer Concern: Why are all experiments conducted only on Qwen3-4B-Instruct-2507? Do the improvements also hold for another backbone?
Response: Following our earlier response, we conducted additional full RL experiments using Granite-4.2-8B, a larger backbone from a different model family, on TextWorld-Sync, to further validate SERA's cross-family generalizability.
| Method | Easy ↑ | Medium ↑ | Hard ↑ | Extreme ↑ | Avg. ↑ |
|---|---|---|---|---|---|
| Granite-4.2-8B | 64.67 ± 0.92 | 19.43 ± 1.31 | 0.00 ± 0.00 | 0.00 ± 0.00 | 21.02 ± 0.51 |
| GRPO | 99.71 ± 0.29 | 65.81 ± 2.29 | 12.95 ± 0.87 | 0.00 ± 0.00 | 44.62 ± 0.86 |
| RAO | 99.43 ± 0.29 | 95.43 ± 1.59 | 38.86 ± 1.87 | 10.29 ± 1.31 | 61.00 ± 0.84 |
| SERA (Ours) | 100.00 ± 0.00 | 99.33 ± 0.33 | 63.81 ± 2.72 | 24.95 ± 1.19 | 72.02 ± 0.39 |
Key findings. SERA's full RL training benefits generalize to a larger, non-Qwen backbone.
- Overall improvement: SERA achieves 72.02% average success, outperforming RAO (61.00%) by +11.02 percentage points.
- Challenging tasks: SERA improves Hard and Extreme success by +24.95 and +14.66 percentage points, respectively.
Your Feedback Would Be Appreciated
Dear Reviewer F4XM,
As the discussion period closes in approximately five hours, we would like to kindly follow up on your concerns.
We have conducted additional experiments on Granite-4.2-8B (+11.02 points over RAO) and Terminal-Bench 2.1 (+11.2 points), directly addressing model and task generalizability.
We hope these results resolve your concerns and would sincerely appreciate your reconsideration of our work.
Thank you for your thoughtful feedback.
Best regards,
Authors
W1: Generalization Across Different Model Backbones
Response: Thank you for this valuable suggestion. To assess the generalizability of SERA across model backbones, we conducted additional experiments using Qwen3.5-4B as a second backbone on the TextCraft benchmark. We evaluated each method over three independent runs to account for randomness and report the mean ± standard deviation.
Key findings. SERA consistently outperforms RAO across all four difficulty levels on Qwen3.5-4B. Specifically:
W2: Generalization Beyond TextCraft-Synth and TextWorld-Sync
Response: Thank you for raising this important question. To investigate whether SERA generalizes beyond TextCraft-Synth and TextWorld-Sync, we conducted an additional evaluation on Terminal-Bench 2.1, a benchmark featuring realistic terminal-based engineering tasks.
Experimental setup. We use DeepSeek-V4.1-Flash-1017 as the base model, using the mini-swe-agent harness with the maximum 256K context. We incorporate SERA's core mechanism—recursive subtask delegation with self-evaluation—into the agent's execution workflow: At each step, the parent policy decides whether to solve the task directly or delegate a focused subtask to a child instance, and then self-evaluates the child’s returned work against the subtask criteria to either accept the result or trigger a retry.
On Terminal-Bench 2.1, adding SERA-style recursive delegation and self-evaluation raises the pass rate from 56.2% to 67.4% (+11.2 points). This shows that in-policy evaluation transfers effectively to realistic terminal engineering tasks and provides a substantial gain through test-time verification and retry.