- Rating
- 4(+1)
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
LLM-driven agents often lose effectiveness when operating in software environments they have not encountered before. In real-world deployments, however, agents are likely to face unfamiliar interfaces, features, and workflows on a regular basis. A conventional approach is to collect additional examples and update the model, but this can become expensive and difficult, particularly when the software is proprietary or frequently changing. Retraining or fine-tuning a frontier-scale LLM may also be prohibitively costly for many practical use cases.
Instead, this paper introduces a test-time learning strategy in which the system develops its own practical guide through interaction with the software. The authors design a multi-agent system consisting of an actor, a verifier, and a curriculum agent. Together, these agents explore the software, propose and conduct experiments, evaluate the resulting outcomes, and record useful observations in reusable memory. During later task execution, the system retrieves relevant information from this memory, allowing it to improve its performance without modifying the underlying model parameters.
The learning process follows a broad-to-deep search strategy. It first surveys many parts of the application to build a general understanding of its functionality and behavior. It then focuses more closely on ambiguous interactions, edge cases, and unusual behaviors that may have been overlooked during the initial exploration. According to the reported results, this approach enables openly available models to match or outperform leading proprietary systems on computer-use agent benchmarks.
Critical strengths
(System design) The paper is very easy to read and follow. The multi-agent system is very well designed (kudos to the authors for a good job!). I really liked the actor, verifier and curriculum design. The authors took great care to design the system. The exact information separation is one that I particularly liked. For example, in the paper’s design, the verifier can’t see the actor's reasoning/memory so as to reduce the chances of correlated errors.
(Good exploration strategies) The broad then deep exploration split is very nice and a real structural contribution. I think this is how humans would also typically learn a new topic/environment, and the authors rightly deserve credit for operationalizing that idea for LLMs at scale.
(Nice ablations showing separation of harness and RSI stage) I like the ablations separating the gains from the RSI stage from their particular harness design. In OSWorld2.0, for example, there is significant gains from RSI (71.97 -> 78.98 partial, 37.80 -> 42.68 binary), aside from the harness itself.
(Generalizes beyond computer use) The fact that the authors went beyond computer use agents to GameCraft-Bench game development is one of the reasons I am impressed by this paper.
(Honest and thorough) The paper is very honest and thorough in its failure mode analysis, both in the main paper and in the appendix. This makes the results more believable.
Critical weaknesses
The paper is generally strong. I can think of a few weaknesses, and I will list them in the order of descending priority. I am happy to increase my score should the authors address the issues mentioned here.
Tone down the claims related to RSI
I think the claim for recursive self-improvement is partly overclaiming. The method here is training free, and the models/prompts/harnesses are fixed. The only thing evolving is an external memory, and I am not sure if that counts as recursive self-improvement. I would urge the authors to not use a catchy phrase for the sake of it. A more honest claim is “autonomous test-time learning through exploration + external memory”.
A further reason I would suggest this is because in this paper’s framework, exploration is performed under the assumption that the target query is known to the curriculum agent. This is more akin to gathering information useful for a particular task. A stronger test would require exploring the environment before revealing the particular evaluation tasks, freeze memory, then test the model on genuinely held-out tasks (unsupervised exploration). I would increase my score if the authors could do one of the following things:
-
Tone down the claims (how is the current thing different from a more fleshed-out reAct or agent loop?)
-
Or provide experiments where the target query is unknown to the curriculum agent
-
Or builds memory via target-conditioned exploration on a bunch of environments, freeze the memory, and test on really OOD tasks (see Paprika [1]).
Budget matched comparison
How does the closed source models do under this paper’s harness and/or if one matches the budget for tokens/tool calls etc. exactly? Could the authors provide a comparison of costs between the baselines and their framework?
Questions for authors
(Fair comparison against meta-RL like approaches)
The paper mentions that training based approaches for adaptation can be difficult to apply/implement. However, a common approach can be training the language models directly on a diverse set of environments that require strategic exploration to succeed, and the models can then generalize on downstream tasks and know how to explore and collect information in their new test-time environments before acting, significantly improving their success rate. In general, a scaled-up approach similar to Paprika [1] comes to mind. Could the authors add a discussion related to this in their paper and provide a short summary of the discussion in their rebuttal? In short, is there a reason to not train the models directly to internalize the gains from their custom harnesses through further training?
(Mention the compute cost)
Despite not requiring any training, the proposed framework must use a substantial amount of compute exploring the new environments and building the memory. Could the authors provide a compute cost (tokens/GPU hours etc.) for their method on each of the benchmarks (with and without RSI)?
References
[1] Training a Generally Curious Agent, https://arxiv.org/abs/2502.17543
7 comments
3. “Budget matched comparison.” (W2)
We compare RSIAgent with repeated attempts, iterative refinement, and reflection memory as alternative ways to use additional compute. All methods share the GLM-5.3 actor agent, execution harness, and Kimi-K3 verifier agent. The five-task comparison continues to RSIAgent-based token and tool-call limits, testing whether RSIAgent’s gains persist under comparable resource use. Actual usage and cost are reported below.
New experiment 2. Same-harness comparison with token and tool-call limits.
A. Selected task set · method-specific stopping
| actor agent | Method | Partial (%) | Binary (%) | Tokens (M) | Tool calls | Cost (USD) |
|---|---|---|---|---|---|---|
| GLM-5.3 | Repeated attempts | 40.41 | 8.33 | 793 | 2,311 | $364 |
| GLM-5.3 | Iterative refinement [R1] | 43.24 | 0.00 | 430 | 2,041 | $205 |
| GLM-5.3 | Reflection memory [R2] | 53.06 | 16.67 | 457 | 2,334 | $206 |
| GLM-5.3 | RSIAgent | 71.11 | 16.67 | 2,900 | 19,600 | $1,160 |
B. Five-task subset · method-specific stopping
| actor agent | Method | Partial (%) | Binary (%) | Tokens (M) | Tool calls | Cost (USD) |
|---|---|---|---|---|---|---|
| GLM-5.3 | Repeated attempts | 42.00 | 20.00 | 193 | 595 | $87 |
| GLM-5.3 | Iterative refinement [R1] | 42.00 | 0.00 | 100 | 666 | $48 |
| GLM-5.3 | Reflection memory [R2] | 48.00 | 20.00 | 93 | 610 | $41 |
C. Same five-task subset · continue to resource limits
| actor agent | Method | Partial (%) | Binary (%) | Tokens (M) | Tool calls | Cost (USD) |
|---|---|---|---|---|---|---|
| GLM-5.3 | Repeated attempts | 50.00 | 20.00 | 1,009 | 6,704 | $493 |
| GLM-5.3 | Iterative refinement [R1] | 42.00 | 0.00 | 1,015 | 7,134 | $494 |
| GLM-5.3 | Reflection memory [R2] | 56.00 | 20.00 | 1,000 | 7,176 | $512 |
| GLM-5.3 | RSIAgent | 76.00 | 40.00 | 1,059 | 7,165 | $423 |
Experimental setup. We use the OSWorld 2.0 tasks from the query-blind experiment. All methods see the target query and share the GLM-5.3 actor agent, execution harness, and Kimi-K3 verifier agent. Token and tool-call limits follow RSIAgent’s measured usage on the five tasks with logs and the full-set mean on the remaining tasks. Official scores never guide stopping or candidate selection; usage includes all agent roles and discarded attempts. We will add the full experimental details to the revision.
Results and analysis. The same-harness w/o RSI baseline scores 26.29%. With their own stopping rules, TTS baselines reach 40.41–53.06%, versus 71.11% for RSIAgent (+18.05 pp over the strongest baseline). To test whether early stopping explains the gap, we also continue the baselines to the resource limits on the five-task subset. They consume 1,000–1,015M tokens, close to RSIAgent’s 1,059M, but reach at most 56.00% versus 76.00% (+20.00 pp). This 20.00 pp advantage supports the contribution of the exploration–memory loop under comparable resource use.
4. “Fair comparison against meta-RL like approaches” (Q1)
Paprika trains general exploration skills across tasks and transfers them to unseen tasks [R3]. This is complementary to learning environment-specific memory at deployment. Training can internalize reusable strategies, while RSIAgent can adapt without weight updates. A trained exploration policy could also serve as the actor agent in RSIAgent. We will add more detailed discussion in the revision.
5. “Mention the compute cost” (Q2)
The comparison above reports token usage, tool calls, and cost. Baseline costs are provider-billed; RSIAgent costs are estimated assuming a 97% cache-hit rate. Both exclude the official evaluator and user simulator. We will add these accounting details to the revision.
References
[R1] Madaan et al. (2023). Self-Refine: Iterative Refinement with Self-Feedback. NeurIPS.
[R2] Shinn et al. (2023). Reflexion: Language Agents with Verbal Reinforcement Learning. NeurIPS.
[R3] Tajwar et al. (2025). Training a Generally Curious Agent (Paprika). ICML.
Kudos to the authors for a thorough rebuttal. I have a few additional questions.
(Question 1) Query blind vs Query aware RSI.
Is there any qualitative + quantitative analysis the authors can provide in terms of how different the generated memories are, when the curriculum agent knows the target query vs when it does not? How do they differ?
(Question 2) Specific question about new experiment 1
Documents & email 0.00 0.00 60.00
Is there a reason the query blind RSIAgent is so poor in this scenario? How many test scenarios are there in this category?
Dear Reviewer BMx3,
Thank you for your detailed review and for recognizing the system design and exploration strategy. Your suggestions helped us clarify the scope of our RSI claim.
1. “Tone down the claims related to RSI” (W1 - Part 1)
Thank you for raising this point. Let us clarify: RSIAgent implements agent-level recursive self-improvement within the broader test-time scaling framework. Exploration reveals capability gaps; verifier agent feedback guides corrections; and the resulting experience updates persistent memory. The curriculum agent and actor agent then use this updated state to better guide subsequent practice and execution. The process is recursive because what the agent learns in the current round changes how it explores and improves in the next round.
This dependence between cycles is visible in Appendix E.3. In slide repair, verifier agent feedback reveals a mistaken interpretation of the constraints; memory records the correction, which is tested in subsequent practice and reused by the actor agent to repair the slides. Figure 3 (§4.4, p. 7) shows the performance progression over RSI steps 0–8: watermark removal improves from 40% to 100%, slide repair from 40% to 80%, and rail booking from 0% to 100%. The evolving unit is the complete agent system, including persistent memory and the behavior it conditions.
2. “provide experiments where the target query is unknown to the curriculum agent” (W1 - Part 2)
Following your suggestion, we conducted a query-blind experiment. BRS built fresh memory through broad exploration of the application environment. The curriculum agent then used this memory in DRS to propose a challenging but feasible, comprehensive practice task with verifiable subgoals. The actor agent practiced on it and updated memory using verifier agent feedback. All roles were blind to evaluation queries and target-specific artifacts. Memory was frozen and the environment reset before revealing the query for final execution, using the baseline’s backbone, harness, and execution cap. We will add the experiment and protocol details to the revision.
New experiment 1. Query-blind RSIAgent by domain (partial score, %).
Overall binary accuracy: Baseline 0.00%; w/o query 0.00% on raw official scores (8.33% with the Web date diagnostic); w/ query 16.67%.
Results and analysis. Query-blind RSIAgent reaches 56.09% partial score versus 26.29% for the baseline (+29.80 pp), improving 9 of 10 domains. These results show that autonomous practice can build useful application knowledge before the evaluation task is known.
The spreadsheet case makes this memory-level claim concrete. Practice produced notes on formulas, cached values, and native charts; the actor agent later retrieved them to recalculate stale outputs and verify cell dependencies. Across the spreadsheet domain, the partial score rises from 5.00% to 57.02%. This illustrates how accumulated experience changes subsequent behavior. Query-conditioned exploration further focuses the same learning loop (71.11% w/ query versus 56.09% w/o query overall).