- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The authors introduce RSIE, a closed-loop RSI system over "expert environments."
FrontierWorld constructs executable task packages from already-published research and professional deliverables (released studies with data/code, clinical notes, financial models, engineering reports). Each package consists of an instruction, an assembled workspace, and a task-specific reference rubric; reference material is hidden from the agent, and a static validation certifies package quality without executing a solver. A task state machine (reserve / active / mastered / revisit) refreshes the active frontier when tasks saturate.
Experience admission. A GPT-5.4 diagnoser writes candidate experience entries in a fixed schema (trigger / procedure / validation / repair; four functional views). Eq. 3 admits an entry only when its reuse raises the same task's valid score beyond the previous best under the unchanged evaluator; invalid runs (timeouts, worker failures, contract failures) are excluded.
Internalization. Verified experience is used through (i) externalized retrieval — conservative, transferable domain-skill and general-workflow views only, at most one entry retrieved, never across the task boundary — and (ii) parametric internalization via on-policy self-distillation (LoRA, generalized JSD, λ = 1, ZeRO-3, no distilled answer).
Evaluation. Ten cycles from Qwen3.6-35B and Qwen3.5-9B on ResearchClawBench (600 active tasks), SGIBench (5,000 generated tasks; 318 DeepResearch + 68 wet experiments), and GDPVal (1,000 active tasks; 220 scored), all executed through the common ResearchHarness (4 CPU cores / 8 GB; 2–4 h per-task budgets).
Headline results. 35B: RCB 12.1 → 20.1, Wet 23.37% → 56.90%, DeepResearch 48.43% → 59.50%, GDPVal 838 → 1339 Elo. 9B: 8.83 → 14.9, 17.80% → 48.80%, 44.97% → 48.56%, 608 → 825 Elo.
Critical strengths
- Table 1 organizes representative systems by improvement mechanism (task/world, model update, experience reuse, verified retention) and feedback regime (judge, cost per run), cleanly positioning the paper at the "hours per run, external rubric" corner that existing RSI work does not cover.
- Admission requires strict same-task improvement over the agent's own prior best under the unchanged evaluator; infrastructure failures are dropped rather than promoted; the static validation cannot roll out a solver and hence cannot certify solvability; the public/hidden boundary ensures the agent never reads the reference reports. These are the correct instincts for a setting with sparse positive feedback.
- Experience organization (four functional categories, transferable vs. task-bound, with the 56.7%/87.0% skill-overlap diagnostic), the checklist variant (18.7), progressive exposure vs. one-shot injection (19.2 vs. 16.9), fixed vs. refreshed frontier (best 19.2; cycle-10 18.7), data scaling (N = 64…600), and an offline SFT control accompanied by a credible trajectory audit (1,000 → 348 retained examples; 87.3% repeated-skill overlap), plus a DPO control with a frozen base agent and disjoint training pools.
- Benchmarks are never averaged; native evaluators and formats are retained; the wet subset is kept separate; the authors decline to fit a scaling law to four measured points; the lack of an SFT validation split and the shared diagnoser/judge model are disclosed rather than hidden.
Critical weaknesses
- GPT-5.4 is simultaneously the experience extractor and the judge for three of four metrics. I am therefore not able to attribute the reported gains to improved expert-task competence rather than to improved alignment with one model family's rubric.
- No RSI method is included as an experimental baseline.
- Pretraining contamination of the published-deliverable source is not examined. FrontierWorld is built from real, already-released documents (released studies, clinical notes, financial models, reports). The public/hidden boundary protects the reference during a run, but the Qwen base models may already have encountered the source deliverables during pretraining. There is no source-to-pretraining overlap analysis (titles, URLs, n-grams, citations) and no memorization control, and no analysis of whether high scores reflect prior knowledge of the source text rather than executed competence. Held-out-by-task-identifier (600 / 5,000 / 1,000 active pools versus 40 / 386 / 220 scored items) is not held-out-by-task-family.
- Cross-task-family transfer rests only on aggregate benchmark totals. The conclusion that improvement "transfers across task types" is supported by benchmark totals, not by a transfer test. Retrieval is deliberately conservative (one entry, transferable views only), but no experiment evaluates on a task family entirely unseen during experience accumulation; no endpoint retrieval-hit-rate or counterfactual (retrieval disabled) breakdown is given; and the 56.7% skill-overlap number is a text-statistic proxy for transfer, not behavioral evidence of transfer.
- No total compute/cost accounting and no uncertainty reporting. Ten cycles over 600 / 5,000 / 1,000 active tasks, with 2–4 h per-task budgets and pervasive GPT-5.4 judging, warrant a total compute ledger (CPU/GPU hours, GPT-5.4 calls and tokens, estimated cost, Elo pairwise-comparison counts); none is reported. Results appear to be single runs: no seeds, no standard deviations, no confidence intervals, and no CIs or bootstrap intervals for the Elo estimates, despite post-hoc "best checkpoint" selection (e.g., 19.2 at cycle 5).
Questions for authors
- Please re-score all three benchmarks using judges from a different model family (ideally two; e.g., Claude, Gemini, or DeepSeek), blind to the training condition, and report (i) inter-judge agreement (Cohen's κ or pairwise agreement) against GPT-5.4, (ii) the complete score tables under the alternative judges, and (iii) whether the gains and rank orderings survive. A human-rated subset as an anchor would further strengthen the claim. Does the wet-experiment subset remain the only independent signal, and do gains hold there with more tasks?
- Please (i) define the exact configuration behind Table 3's "RSIE" (19.2) and Table 4's N = 600 (19.20) — model size, retrieval on/off, frontier refresh, checkpoint selection — and state how they relate to the complete 20.1 RSIE in Table 2; (ii) reprint Tables 3 and 4 with model size and retrieval flags on every row and explicitly discuss the 19.20 (35B, w/o retrieval) vs. 14.9 (9B, complete) comparison; (iii) restore the wet values in §4.2 or state explicitly that the column was omitted; (iv) correct the Table 9 header and confirm the row-to-model mapping; (v) state architecture, total/activated parameters, and checkpoint IDs for both bases, including whether "35B" is a MoE active-parameter count.
- What is the admission gate's measured gain on task families never seen during experience accumulation (e.g., extract experience on two domains/benchmarks and evaluate on a third, with no extraction from it)? Please report retrieval hit rates at the endpoints, a per-task counterfactual with retrieval disabled, and transfer broken down by task family rather than only by benchmark totals. 4 Please adapt at least one RSI method (R-Zero, ReasoningBank, or KOPE) into ResearchHarness under matched task budgets and the same native evaluators, on at least one benchmark. It need not win; the point is to make the claimed "expensive setting" contrast experimentally testable.
- Please provide (i) a contamination audit of the source deliverables against pretraining (title/URL/n-gram overlap and memorization probes) and (ii) a full compute/cost ledger (CPU/GPU hours, GPT-5.4 calls/tokens, Elo pair counts); repeat the headline runs with multiple seeds and report standard deviations or confidence intervals for the Elo estimates.
6 comments
Answer of Q2 To Reviewer 8jNP
Q2. Clarification of experimental settings, table labels, and model details
The scores of 19.2 in Table 3 and 19.20 in Table 4 both correspond to Qwen3.6-35B-A3B without external experience at inference time, matching the “w/o experience” configuration in Table 2. The score of 20.1 corresponds to the full RSIE configuration using the same model with external experience enabled. We have standardized model names and configuration labels across these tables and added per-round wet-experiment results and model details.
1. The scores of 19.2 and 20.1 reflect different inference configurations of the same model.
The RSIE reference row in Table 3 and the Qwen3.6-35B-A3B result at in Table 4 both follow the RSIE training pipeline, including FrontierWorld task refreshing, and use checkpoint-2000, obtained after 2,000 post-training steps. External experience is disabled at inference time in both cases, yielding the same score of 19.2 reported in Table 2.
Here, “experience” refers to external guidance loaded from the experience bank at inference time. The “w/o experience” setting disables only this inference-time guidance; experience extraction, validation, and OPSD updates remain part of training. Enabling external experience for the same model increases the RCB score from 19.2 to 20.1, indicating that the experience bank provides additional benefits beyond the capabilities acquired through post-training.
2. Tables 3–4 now include complete model names and an Experience label for each row.
All rows in Table 3 use Qwen3.6-35B-A3B. Table 4 reports results for Qwen3.6-35B-A3B and Qwen3.5-9B. The Experience column is marked “w/o” throughout both tables.
Qwen3.6-35B-A3B achieves 19.20 without external experience, exceeding the 14.9 achieved by Qwen3.5-9B under the full configuration. However, this comparison varies both the model and the Experience setting, so it cannot isolate the contribution of external experience. The corresponding within-model comparisons are 19.2 → 20.1 (+0.9) for Qwen3.6-35B-A3B and 13.1 → 14.9 (+1.8) for Qwen3.5-9B. Both comparisons support the additional benefit of external experience.
3. Appendix A.2 now includes complete per-round results, including deterministic wet-experiment scores.
Table 6 in Appendix A.2 of the revised manuscript reports ten rounds of evaluation for Qwen3.6-35B-A3B on RCB, GDPVal, SGI-DR, and SGI-Wet. The wet-experiment evaluation uses the “w/o experience” configuration and follows the benchmark’s deterministic scoring rules.
The scores across the ten rounds are 23.37, 25.66, 25.91, 27.02, 30.72, 35.77, 41.62, 48.83, 52.21, and 50.01. Performance peaks at 52.21 in round 9, matching the result for this configuration in Table 2. The round-10 score is 50.01, still 26.64 points above the first round. These results show that improvements also hold under deterministic evaluation without an LLM judge, while revealing performance fluctuations in later rounds. The full RSIE score of 56.90 in Table 2 uses external experience and therefore corresponds to a different inference configuration.
4. Table 9 in the original manuscript, now Table 12, includes complete model names and explicit score labels.
The revised table separately reports the base-model score, the score after SFT, and the corresponding change. The mapping between models and scores remains unchanged.
5. We have added complete model names, parameter counts, and evaluation checkpoint details.
Qwen3.6-35B-A3B uses an MoE architecture with 35B total parameters and 3B active parameters per token. Qwen3.5-9B is a dense model with 9B parameters. Their base checkpoints come from the official repositories Qwen/Qwen3.6-35B-A3B and Qwen/Qwen3.5-9B, respectively. RSIE evaluations in this paper use checkpoint-2000, obtained after 2,000 post-training steps.
Answer of Q3 To Reviewer 8jNP (Part 1/2)
Q3. Cross-domain transfer of validated skills, retrieval coverage, and comparison with RSI methods
We thank the reviewer for this valuable suggestion. We provide additional evidence through two routes: parameter internalization and external experience reuse. After experience accumulation and OPSD training exclusively on Math tasks, the model improves by 5.9 and 7.0 points on the fully held-out Life and Earth domains, respectively, without external experience at test time. In the standard RCB evaluation, external experience is loaded for every task, and enabling it further improves the average score in all ten domains. RSIE also achieves the highest score in our additional comparison with representative RSI methods on expert research tasks, outperforming the strongest baseline by 5.0 points.
1. Validated experience supports capability improvement in domains excluded from learning.
Following the reviewer’s suggested setting of extracting experience from some domains and evaluating on others, we started from the same Qwen3.6-35B-A3B base model and selected 98 Math-related tasks within the RSIE pipeline. Only execution feedback from these tasks was used to extract and validate experience, and only experience passing the admission criterion was used for OPSD parameter updates. Life and Earth seeds were excluded from training task retrieval, and all tasks derived from those seeds were excluded from experience extraction, selection, and parameter updates.
Evaluation used the original official RCB test tasks and evaluation protocol, with all test tasks kept separate from training tasks. External experience was disabled at test time to directly assess cross-domain capabilities retained in the model parameters after OPSD training.
| Model / Training setting | Math ↑ | Life ↑ | Earth ↑ |
|---|---|---|---|
| Qwen3.6-35B-A3B base model | 10.0 | 8.5 | 15.0 |
| RSIE trained only on selected Math tasks | 22.5 | 14.4 | 22.0 |
| Improvement | +12.5 | +5.9 | +7.0 |
Although expert research tasks involve different domain knowledge, they share requirements such as planning analytical steps, checking assumptions, validating results, and diagnosing and correcting errors based on feedback. These processes provide learning content that can be reused across domains. Life and Earth both improve despite being excluded from experience accumulation and receiving no external experience at test time. The setup and results of this held-out domain experiment have been added to Appendix A.5.3, Table 11 of the revised manuscript.
2. External experience is retrieved and loaded for 100% of tasks in the final RCB evaluation.
When experience is enabled, each test task retrieves and loads the highest-similarity eligible entry from the experience bank. Thus, defining the loading rate as the proportion of test tasks for which experience is actually loaded, coverage is 100% both overall and within each of the ten RCB domains.
This rate describes the coverage of external experience. To assess its actual contribution, we additionally compare the same model with and without external experience. The retrieval settings have been clarified in Appendix A.5.2.
3. Comparisons on the same tasks show additional benefits from external experience across all ten RCB domains.
As clarified in our response to Q2, 19.2 and 20.1 correspond to the same trained model with external experience disabled and enabled, respectively. We fixed the model checkpoint and evaluation settings, executed both configurations on the same test tasks, and aggregated results over the ten original RCB domains, each containing four tasks.
| RCB domain | Base model | RSIE w/o experience | RSIE w/ experience | Gain from external experience |
|---|---|---|---|---|
| Astronomy | 23.7 | 32.9 | 34.7 | +1.8 |
| Chemistry | 3.0 | 9.6 | 10.3 | +0.7 |
| Earth | 15.0 | 29.0 | 30.6 | +1.6 |
| Energy | 18.4 | 24.4 | 25.5 | +1.1 |
| Information | 5.5 | 12.9 | 13.3 | +0.4 |
| Life | 8.5 | 9.7 | 11.1 | +1.4 |
| Material | 12.3 | 18.9 | 19.2 | +0.3 |
| Math | 10.0 | 22.0 | 22.3 | +0.3 |
| Neuroscience | 1.1 | 6.8 | 7.2 | +0.4 |
| Physics | 23.5 | 25.8 | 26.8 | +1.0 |
| Mean | 12.1 | 19.2 | 20.1 | +0.9 |
This table reports the standard RCB evaluation after full RSIE training, separately from the Math-only held-out domain experiment in point 1. Without external experience, RSIE already outperforms the base model in all ten domains. Enabling experience further improves domain-level average scores by 0.3–1.8 points. These results break down the overall gains by domain and support the complementary roles of parameter learning and external experience. The complete comparison has been added to Appendix A.5.2.
Answer of Q1 To Reviewer 8jNP
We thank the reviewer for this valuable suggestion.
Q1. Reliability of evaluation across LLM judges, human experts, and deterministic scoring
RSIE consistently outperforms the base model and SFT across all three benchmarks when evaluated by alternative judges, with the method ranking unchanged. Independent evaluations by two human experts also confirm the improvement. In addition, after expanding the wet-experiment evaluation to 200 tasks, RSIE retains a 35.62 percentage-point gain over the base model under deterministic scoring. Below, we provide evidence from these three perspectives.
1. Consistent improvements across judge families.
We rescored the same final artifacts using DeepSeek-V4.1-Flash and Gemini-3.8-Flash, keeping the tasks, original evaluation criteria, and score aggregation procedures unchanged. Only the judge model was replaced. Method names and training conditions were hidden, and the presentation order of paired artifacts was randomized. All results below use Qwen3.6-35B:
All three judges preserve the ranking RSIE > SFT > base model on all three benchmarks. Relative to the base model under each judge, RSIE improves RCB scores by 7.0–8.0 points, SGI-DR scores by 10.32–11.30 percentage points, and GDPVal ratings by 484–501 Elo.
We further measure pairwise agreement by comparing each alternative judge’s win/loss/tie decisions with those of GPT-5.4 on base-model and RSIE artifacts from the same tasks. Agreement is the proportion of decisions that match exactly.
These judge families produce similar scores, preserve the same method ranking, and agree on relative output quality for most tasks. Together, these results support the improvement in expert task performance beyond the scoring preferences of GPT-5.4.
2. Independent human evaluation confirms improved artifact quality.
We included all RCB test tasks in a human evaluation conducted by two experts with relevant research backgrounds. Each expert independently evaluated the code, analysis results, and reports produced by the base model and RSIE, following the original checklist and its weights. Task inclusion did not depend on model scores. Method identities were hidden, artifact order was randomized, and the experts did not discuss their assessments before submitting scores.
Although the experts differ in their absolute scores, both rate RSIE above the base model, with a mean improvement of 7.5 points. This provides independent human evidence of improved artifact quality, complementing the wet-experiment evaluation as evidence that does not rely on an LLM judge.
3. Gains persist under deterministic scoring on an expanded task set.
The SGIBench wet-experiment evaluation uses deterministic action-scoring rules and does not rely on an LLM judge. We expanded the evaluation from 68 to 200 tasks while retaining the same scoring procedure:
On the expanded set of 200 tasks, RSIE improves over the base model by 35.62 percentage points, confirming that the gains persist across a larger set of wet-experiment tasks. Taken together, evaluation across judge families, independent expert assessment, and deterministic scoring support the interpretation that the observed gains reflect improved expert task performance beyond adaptation to a single judge’s preferences.