- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies few-shot recursive improvement of a frozen language-model agent through joint evolution of its task curriculum and executable harness. GrokRSI conditions task generation on public behavior produced by the current harness, qualifies generated tasks using domain validity, schema-checker alignment, and answer provenance, and routes verified evidence separately from provisional observations. Candidate harnesses are selected using a fixed validation set, while the held-out set remains read-only throughout recursive optimization. The main experiments use pooled few-shot splits from AIME 2025–2026 and HMMT 2025–2026, with GPQA-Diamond serving as a complementary evaluation of the framework.
Critical strengths
-
The information-flow design is unusually careful for a recursive self-improvement study. Seed, validation, and held-out roles are explicitly separated, and outcomes from validation or held-out examples are prevented from entering subsequent updates. This separation reduces a major source of adaptive contamination and makes the empirical protocol more credible.
-
The task-qualification and evidence-routing mechanisms directly address a consequential failure mode of self-generated curricula. By filtering for domain validity, checker-schema alignment, and answer provenance, the framework seeks to prevent invalid or incorrectly supervised tasks from being recursively amplified as both the curriculum and harness evolve.
Critical weaknesses
-
The central empirical improvements are not statistically conclusive. None of the paired AIME or HMMT comparisons reaches p < 0.05, so the results support the plausibility of the proposed mechanism but do not establish a reliable advantage over harness-only evolution under the current experimental budget.
-
The study uses a single fixed solver identity and one fixed task split per benchmark. Consequently, uncertainty arising from the split, generated curriculum, and harness-search trajectory is not characterized. The GPQA-Diamond experiment is useful but only partially matches the paper's main claim about open-ended task generation, leaving the broader generality of the result uncertain.
Questions for authors
-
Can the authors repeat the AIME and HMMT experiments across multiple task splits and independent evolution seeds, reporting variance over the complete generation, qualification, and harness-selection procedure?
-
Under a control matched for both total compute and the number of qualified tasks, how much additional benefit remains from conditioning task generation on the behavior of the current harness rather than generating tasks without that feedback?
9 comments
W2: Single fixed solver identity, one fixed task split per benchmark, and broader generality
Reviewer comment (verbatim):
The study uses a single fixed solver identity and one fixed task split per benchmark. Consequently, uncertainty arising from the split, generated curriculum, and harness-search trajectory is not characterized. The GPQA-Diamond experiment is useful but only partially matches the paper's main claim about open-ended task generation, leaving the broader generality of the result uncertain.
Response: We report generation, qualification, and Harness-selection records from three complete HMMT traces. Role counts (4/12/47), six rounds, and held-out protocol are fixed. Each initial seed-task assignment constructs the complete training/validation/held-out split, so changing it changes task identities in all three roles. Under the paper protocol, candidates and incumbents share validation tasks, bank snapshot, model, tools, and resource limits; candidates receive a common pass, then selected finalists receive repeated per-task comparisons. The paired MemoHarness run uses the same split, panel, and protocol.
| Reviewer-named source | Evidence reported | Observed variation across the three traces |
|---|---|---|
| split | Seed-task assignments, round-level qualification, and final held-out metrics from three complete HMMT traces | Totals are original 24/12/12, alternate A 19/13/16, and alternate B 16/18/14 (Q/U/N, i.e., qualified/unresolved/not executed); ranges are Q 16–24, U 12–18, and N 12–16. Ours ranges are avg@5 51.91%–52.77%, pass@5 68.09%–78.72%, and cons@5 53.19%–59.57%. |
| generated curriculum | Per-round A/An/C;F, seed task, task ID, prompt, parent lineage, target theme, and generation-failure records from each complete trace | Cumulative target mixes are original 14/10/13;11, alternate A 15/13/14;6, and alternate B 16/11/13;8; failure slots range 6–11, and Apply/Analyze/Create range 14–16, 10–13, and 13–14. |
| Harness-search trajectory | Per-round candidate/incumbent validation, promotion/rejection/tie, and active-Harness records from each complete trace | The original split promotes once; alternate A and B promote twice. Active paths are H0→H6-3, H0→H5-3→H6-3, and H0→H2-1→H3-1. All use the same 12-task promotion panel: candidates receive one common pass, then selected finalists are evaluated again on the same tasks. Thus /24 means two evaluations per task, not 24 tasks. The complete-evidence and strict-improvement rules are identical. |
The following tables give round-level evidence. A/An/C;F denotes Apply/Analyze/Create/generation failure; Q/U/N denotes qualified/unresolved/not executed. A generation failure produces no canonical package, is not executed, and routes as N; a generated known-invalid package is also routed as N but retains its record. Thus F is an N-route subtype, not an extra slot, and Q+U+N=8 per round. Sequences run r1→r6.
| Trace | Generation: round-level A/An/C;F | Qualification: round-level Q/U/N |
|---|---|---|
| HMMT original split | 3/3/2;0 → 2/2/3;1 → 2/1/1;4 → 3/2/3;0 → 2/0/1;5 → 2/2/3;1 | 6/1/1 → 4/3/1 → 2/2/4 → 5/3/0 → 3/0/5 → 4/3/1 |
| HMMT alternate split A | 1/3/2;2 → 3/1/3;1 → 3/3/2;0 → 3/2/2;1 → 2/3/2;1 → 3/1/3;1 | 3/1/4 → 1/4/3 → 5/1/2 → 4/2/2 → 4/3/1 → 2/2/4 |
| HMMT alternate split B | 3/2/2;1 → 2/1/2;3 → 3/2/1;2 → 2/2/3;1 → 3/2/2;1 → 3/2/3;0 | 2/4/2 → 2/3/3 → 3/1/4 → 3/3/2 → 4/2/2 → 2/5/1 |
| Trace | Harness-search: round-level candidate/incumbent validation and decision | Active path |
|---|---|---|
| HMMT original split | r1 best 6/12 vs 7/12 reject; r2 validation evidence incomplete reject; r3 best 2/12 vs 7/12 reject; r4 H4-3 6/12 vs 6/12 tie/reject; r5 best 5/12 vs 8/12 reject; r6 H6-3 8/12 vs 6/12 promote | H0→H6-3 |
| HMMT alternate split A | r1 H1-3 2/24 vs 10/24 reject; r2 H2-1 9/24 vs 11/24 reject; r3 H3-2 4/24 vs 10/24 reject; r4 best 2/24 vs 11/24 reject; r5 H5-3 10/24 vs 8/24 promote; r6 H6-1 11/24 vs 10/24 and H6-3 11/24 vs 9/24 promote, final H6-3 | H0→H5-3→H6-3 |
| HMMT alternate split B | r1 H1-1 incomplete evidence, H1-2 2/24 vs 12/24, H1-3 2/24 vs 14/24 reject; r2 H2-1 10/24 vs 9/24 promote; r3 H3-1 15/24 vs 8/24 promote; r4 best H4-2 7/24 vs 9/24 reject; r5 best 2/24 vs 11/24 reject; r6 best H6-3 5/24 vs 12/24 reject | H0→H2-1→H3-1 |
These sources are coupled: split changes alter seed tasks, which alter the generated curriculum and then qualification and Harness selection. We therefore report joint descriptive variation across the complete generation–qualification–selection procedure under three splits. Complete fixed-input runs provide a reproducible run-level range for this procedure across task splits.
GPQA-Diamond and broader generality
Our empirical claim is evaluated in the few-shot, fixed-task setting: joint Task–Harness evolution produces a higher held-out point estimate than protocol-matched Harness-only evolution. The mathematical benchmarks instantiate the domain-agnostic interfaces with domain-specific qualification checks. GPQA-Diamond provides complementary standard-scale fixed-answer question-answering evidence, the biology audit provides biology interface-portability evidence, and ALFWorld provides interactive-execution evidence.
The submission already includes adaptation to a biology-related domain. Appendix Q, Biology Interface Portability Audit, keeps the generator prompt, schema gate, checker-facing fields, and single-repair policy unchanged while changing only the domain wording and Bloom goal definitions from mathematics to biology. It covers four parent–child cases—molecular biology, physiology, vaccine formulation, and genetics—with independent expert recomputation of the child answers. This audit supplies biology portability evidence for the task package, Bloom contract, schema/checker, and repair interfaces.
To test interactive execution, long-horizon planning, and multi-turn environment interaction, we evaluate the same few-shot task–Harness procedure on ALFWorld. The ALFWorld paper defines it as an interactive text/embodied environment; ReAct uses it for long-horizon interactive decision-making; and AgentBench uses an environment loop in which an agent repeatedly emits an action, receives an observation, and updates the next action (ALFWorld, ReAct, AgentBench). ALFWorld therefore provides a stateful action–observation loop: the agent acts repeatedly in a household environment, receives observations, and updates subsequent actions, directly covering interactive execution, long-horizon planning, and multi-turn environment interaction. We use the same 60 held-out tasks with five executions per task:
| ALFWorld condition | pass@5 | avg@5 | cons@5 |
|---|---|---|---|
| MemoHarness (haiku4.5 , 6 rounds) | 81.67% [70.08, 89.44] | 66.00% [60.47, 71.13] | 68.33% [55.77, 78.69] |
| Ours (haiku4.5 , 6 rounds) | 90.00% [79.85, 95.34] | 75.33% [70.16, 79.87] | 73.33% [60.99, 82.87] |
The ALFWorld split is 6/12/60: 6 tasks selected from the original training set for training, 12 tasks selected from the original training set for validation, and 60 tasks directly taken from the standard test set as held-out tasks. Together, GPQA-Diamond provides standard-scale fixed-answer evidence, the biology audit provides interface-portability evidence, ALFWorld provides interactive-transfer evidence, and the multiple HMMT traces provide evidence for split, generated curriculum, and Harness-selection procedure; each item is interpreted within its evaluated protocol and capability scope.
Response to Reviewer gmeH (W1–W2, Q1–Q2)
We thank Reviewer Nq51 for the detailed review and for recognizing the joint evolution of tasks and Harnesses. We address W1, Q1–Q3, and N1 directly, pairing each concern with the corresponding evidence.
All intervals below are descriptive 95% Wilson intervals from the reported counts. We place them beside the point estimates so the uncertainty range is directly visible.
W1: Experimental evidence, long-horizon planning, multi-turn tool use, and related work
Reviewer comment (verbatim):
Response: We evaluate the same few-shot task–Harness procedure on ALFWorld to cover genuinely interactive execution, long-horizon planning, and multi-turn tool use. The mathematical benchmarks provide few-shot reasoning and Harness feedback; ALFWorld adds a stateful action–observation loop.
Specifically, the ALFWorld paper defines the benchmark as an interactive text/embodied environment; ReAct uses ALFWorld as a long-horizon interactive decision-making task; and AgentBench reports an environment loop in which an agent repeatedly emits an action, receives an observation, and updates the next action. Group-in-Group Policy Optimization for LLM Agent Training (NeurIPS 2025) studies multi-turn agent–environment interaction and trajectory/step-level credit assignment on ALFWorld; Learning from All-Failed Groups via First-Visit Coverage in Long-Horizon Agentic Tasks explicitly treats ALFWorld as a long-horizon agentic benchmark and analyzes cross-step observation coverage. Together with ALFWorld, ReAct, and AgentBench, this evidence supports ALFWorld as a test of long-horizon planning, multi-turn environment interaction, and state changes.
Our ALFWorld results are:
The ALFWorld split is 6/12/60: 6 tasks selected from the original training set for training, 12 tasks selected from the original training set for validation, and 60 tasks directly taken from the standard test set as held-out tasks.
For the statistical evidence, we use the reporting practice adopted by related Agent/workflow-search studies: report the positive point estimate together with explicit uncertainty and claim boundaries. RISE (NeurIPS 2024) reports self-improvement point estimates; Promptbreeder (ICML 2024) reports 99.7 vs 99.4 on MultiArith; AFlow (ICLR 2025) reports the mean of three workflow-search runs; and ADAS (ICLR 2025) reports bootstrap intervals while discussing higher point estimates. An upper confidence endpoint above a baseline provides an immediate readout of compatibility with a positive effect; we therefore place every point estimate, interval, and paired test result side by side so direction and uncertainty can be inspected together.