acceptodds
Under review as a conference paper at ICLR 2027

RSIAgent: Autonomous Exploration for Recursive Self-improvement in New Environments

11 views

Abstract

Digital agents must often adapt to new environments whose interfaces, tools, and failure modes are not fully captured by pretrained models. We introduce RSIAgent, a training-free multi-agent framework for recursive self-improvement through autonomous memory construction. RSIAgent coordinates curriculum, actor, and verifier agents to continually explore the environment, validate outcomes, and retain environment-specific knowledge, including reusable causal relationships between actions, conditions, and consequences. It further adopts a broad-then-deep exploration strategy, combining parallel broad recursive self-exploration for discovering diverse environment structures with focused deep self-exploration for uncovering hard cases, hidden constraints, boundary conditions, and previously unknown causal dependencies. The resulting memory is frozen and can be directly reused for downstream tasks without updating model parameters. Experiments on OSWorld 2.0 and Agents’ Last Exam show that RSIAgent substantially improves strong open-source models, enabling Kimi-K3 and GLM-5.3 to outperform frontier closed-source models including GPT-6 Astra.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Recursive Self-Improvement, Digital Agents, Broad & Deep Self-Exploration, Agentic Causal Discovery

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_BMx3OpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

LLM-driven agents often lose effectiveness when operating in software environments they have not encountered before. In real-world deployments, however, agents are likely to face unfamiliar interfaces, features, and workflows on a regular basis. A conventional approach is to collect additional examples and update the model, but this can become expensive and difficult, particularly when the software is proprietary or frequently changing. Retraining or fine-tuning a frontier-scale LLM may also be prohibitively costly for many practical use cases.

Instead, this paper introduces a test-time learning strategy in which the system develops its own practical guide through interaction with the software. The authors design a multi-agent system consisting of an actor, a verifier, and a curriculum agent. Together, these agents explore the software, propose and conduct experiments, evaluate the resulting outcomes, and record useful observations in reusable memory. During later task execution, the system retrieves relevant information from this memory, allowing it to improve its performance without modifying the underlying model parameters.

The learning process follows a broad-to-deep search strategy. It first surveys many parts of the application to build a general understanding of its functionality and behavior. It then focuses more closely on ambiguous interactions, edge cases, and unusual behaviors that may have been overlooked during the initial exploration. According to the reported results, this approach enables openly available models to match or outperform leading proprietary systems on computer-use agent benchmarks.

Critical strengths

(System design) The paper is very easy to read and follow. The multi-agent system is very well designed (kudos to the authors for a good job!). I really liked the actor, verifier and curriculum design. The authors took great care to design the system. The exact information separation is one that I particularly liked. For example, in the paper’s design, the verifier can’t see the actor's reasoning/memory so as to reduce the chances of correlated errors.

(Good exploration strategies) The broad then deep exploration split is very nice and a real structural contribution. I think this is how humans would also typically learn a new topic/environment, and the authors rightly deserve credit for operationalizing that idea for LLMs at scale.

(Nice ablations showing separation of harness and RSI stage) I like the ablations separating the gains from the RSI stage from their particular harness design. In OSWorld2.0, for example, there is significant gains from RSI (71.97 -> 78.98 partial, 37.80 -> 42.68 binary), aside from the harness itself.

(Generalizes beyond computer use) The fact that the authors went beyond computer use agents to GameCraft-Bench game development is one of the reasons I am impressed by this paper.

(Honest and thorough) The paper is very honest and thorough in its failure mode analysis, both in the main paper and in the appendix. This makes the results more believable.

Critical weaknesses

The paper is generally strong. I can think of a few weaknesses, and I will list them in the order of descending priority. I am happy to increase my score should the authors address the issues mentioned here.

Tone down the claims related to RSI

I think the claim for recursive self-improvement is partly overclaiming. The method here is training free, and the models/prompts/harnesses are fixed. The only thing evolving is an external memory, and I am not sure if that counts as recursive self-improvement. I would urge the authors to not use a catchy phrase for the sake of it. A more honest claim is “autonomous test-time learning through exploration + external memory”.

A further reason I would suggest this is because in this paper’s framework, exploration is performed under the assumption that the target query is known to the curriculum agent. This is more akin to gathering information useful for a particular task. A stronger test would require exploring the environment before revealing the particular evaluation tasks, freeze memory, then test the model on genuinely held-out tasks (unsupervised exploration). I would increase my score if the authors could do one of the following things:

  1. Tone down the claims (how is the current thing different from a more fleshed-out reAct or agent loop?)

  2. Or provide experiments where the target query is unknown to the curriculum agent

  3. Or builds memory via target-conditioned exploration on a bunch of environments, freeze the memory, and test on really OOD tasks (see Paprika [1]).

Budget matched comparison

How does the closed source models do under this paper’s harness and/or if one matches the budget for tokens/tool calls etc. exactly? Could the authors provide a comparison of costs between the baselines and their framework?

Questions for authors

(Fair comparison against meta-RL like approaches)

The paper mentions that training based approaches for adaptation can be difficult to apply/implement. However, a common approach can be training the language models directly on a diverse set of environments that require strategic exploration to succeed, and the models can then generalize on downstream tasks and know how to explore and collect information in their new test-time environments before acting, significantly improving their success rate. In general, a scaled-up approach similar to Paprika [1] comes to mind. Could the authors add a discussion related to this in their paper and provide a short summary of the discussion in their rebuttal? In short, is there a reason to not train the models directly to internalize the gains from their custom harnesses through further training?

(Mention the compute cost)

Despite not requiring any training, the proposed framework must use a substantial amount of compute exploring the new environments and building the memory. Could the authors provide a compute cost (tokens/GPU hours etc.) for their method on each of the benchmarks (with and without RSI)?

References

[1] Training a Generally Curious Agent, https://arxiv.org/abs/2502.17543

7 comments

Dear Reviewer BMx3,

Thank you for your detailed review and for recognizing the system design and exploration strategy. Your suggestions helped us clarify the scope of our RSI claim.

1. “Tone down the claims related to RSI” (W1 - Part 1)

Thank you for raising this point. Let us clarify: RSIAgent implements agent-level recursive self-improvement within the broader test-time scaling framework. Exploration reveals capability gaps; verifier agent feedback guides corrections; and the resulting experience updates persistent memory. The curriculum agent and actor agent then use this updated state to better guide subsequent practice and execution. The process is recursive because what the agent learns in the current round changes how it explores and improves in the next round.

This dependence between cycles is visible in Appendix E.3. In slide repair, verifier agent feedback reveals a mistaken interpretation of the constraints; memory records the correction, which is tested in subsequent practice and reused by the actor agent to repair the slides. Figure 3 (§4.4, p. 7) shows the performance progression over RSI steps 0–8: watermark removal improves from 40% to 100%, slide repair from 40% to 80%, and rail booking from 0% to 100%. The evolving unit is the complete agent system, including persistent memory and the behavior it conditions.

2. “provide experiments where the target query is unknown to the curriculum agent” (W1 - Part 2)

Following your suggestion, we conducted a query-blind experiment. BRS built fresh memory through broad exploration of the application environment. The curriculum agent then used this memory in DRS to propose a challenging but feasible, comprehensive practice task with verifiable subgoals. The actor agent practiced on it and updated memory using verifier agent feedback. All roles were blind to evaluation queries and target-specific artifacts. Memory was frozen and the environment reset before revealing the query for final execution, using the baseline’s backbone, harness, and execution cap. We will add the experiment and protocol details to the revision.

New experiment 1. Query-blind RSIAgent by domain (partial score, %).

Domain / task typeBaselineRSIAgent w/o queryRSIAgent w/ query
Spreadsheets5.0057.0268.69
Presentations25.0045.0050.00
Documents & email0.000.0060.00
Web interaction0.00100.00100.00
CAD design25.0057.4768.97
Electronic design38.4567.4185.25
Image editing60.0080.0080.00
Video editing50.5063.5091.00
Medical imaging36.0144.2358.36
Visual inspection0.0050.0050.00
Overall26.2956.0971.11

Overall binary accuracy: Baseline 0.00%; w/o query 0.00% on raw official scores (8.33% with the Web date diagnostic); w/ query 16.67%.

Results and analysis. Query-blind RSIAgent reaches 56.09% partial score versus 26.29% for the baseline (+29.80 pp), improving 9 of 10 domains. These results show that autonomous practice can build useful application knowledge before the evaluation task is known.

The spreadsheet case makes this memory-level claim concrete. Practice produced notes on formulas, cached values, and native charts; the actor agent later retrieved them to recalculate stale outputs and verify cell dependencies. Across the spreadsheet domain, the partial score rises from 5.00% to 57.02%. This illustrates how accumulated experience changes subsequent behavior. Query-conditioned exploration further focuses the same learning loop (71.11% w/ query versus 56.09% w/o query overall).

3. “Budget matched comparison.” (W2)

We compare RSIAgent with repeated attempts, iterative refinement, and reflection memory as alternative ways to use additional compute. All methods share the GLM-5.3 actor agent, execution harness, and Kimi-K3 verifier agent. The five-task comparison continues to RSIAgent-based token and tool-call limits, testing whether RSIAgent’s gains persist under comparable resource use. Actual usage and cost are reported below.

New experiment 2. Same-harness comparison with token and tool-call limits.

A. Selected task set · method-specific stopping

actor agentMethodPartial (%)Binary (%)Tokens (M)Tool callsCost (USD)
GLM-5.3Repeated attempts40.418.337932,311$364
GLM-5.3Iterative refinement [R1]43.240.004302,041$205
GLM-5.3Reflection memory [R2]53.0616.674572,334$206
GLM-5.3RSIAgent71.1116.672,90019,600$1,160

B. Five-task subset · method-specific stopping

actor agentMethodPartial (%)Binary (%)Tokens (M)Tool callsCost (USD)
GLM-5.3Repeated attempts42.0020.00193595$87
GLM-5.3Iterative refinement [R1]42.000.00100666$48
GLM-5.3Reflection memory [R2]48.0020.0093610$41

C. Same five-task subset · continue to resource limits

actor agentMethodPartial (%)Binary (%)Tokens (M)Tool callsCost (USD)
GLM-5.3Repeated attempts50.0020.001,0096,704$493
GLM-5.3Iterative refinement [R1]42.000.001,0157,134$494
GLM-5.3Reflection memory [R2]56.0020.001,0007,176$512
GLM-5.3RSIAgent76.0040.001,0597,165$423

Experimental setup. We use the OSWorld 2.0 tasks from the query-blind experiment. All methods see the target query and share the GLM-5.3 actor agent, execution harness, and Kimi-K3 verifier agent. Token and tool-call limits follow RSIAgent’s measured usage on the five tasks with logs and the full-set mean on the remaining tasks. Official scores never guide stopping or candidate selection; usage includes all agent roles and discarded attempts. We will add the full experimental details to the revision.

Results and analysis. The same-harness w/o RSI baseline scores 26.29%. With their own stopping rules, TTS baselines reach 40.41–53.06%, versus 71.11% for RSIAgent (+18.05 pp over the strongest baseline). To test whether early stopping explains the gap, we also continue the baselines to the resource limits on the five-task subset. They consume 1,000–1,015M tokens, close to RSIAgent’s 1,059M, but reach at most 56.00% versus 76.00% (+20.00 pp). This 20.00 pp advantage supports the contribution of the exploration–memory loop under comparable resource use.

4. “Fair comparison against meta-RL like approaches” (Q1)

Paprika trains general exploration skills across tasks and transfers them to unseen tasks [R3]. This is complementary to learning environment-specific memory at deployment. Training can internalize reusable strategies, while RSIAgent can adapt without weight updates. A trained exploration policy could also serve as the actor agent in RSIAgent. We will add more detailed discussion in the revision.

5. “Mention the compute cost” (Q2)

The comparison above reports token usage, tool calls, and cost. Baseline costs are provider-billed; RSIAgent costs are estimated assuming a 97% cache-hit rate. Both exclude the official evaluator and user simulator. We will add these accounting details to the revision.

References

[R1] Madaan et al. (2023). Self-Refine: Iterative Refinement with Self-Feedback. NeurIPS.

[R2] Shinn et al. (2023). Reflexion: Language Agents with Verbal Reinforcement Learning. NeurIPS.

[R3] Tajwar et al. (2025). Training a Generally Curious Agent (Paprika). ICML.

Kudos to the authors for a thorough rebuttal. I have a few additional questions.

(Question 1) Query blind vs Query aware RSI.

Is there any qualitative + quantitative analysis the authors can provide in terms of how different the generated memories are, when the curriculum agent knows the target query vs when it does not? How do they differ?

(Question 2) Specific question about new experiment 1

Documents & email 0.00 0.00 60.00

Is there a reason the query blind RSIAgent is so poor in this scenario? How many test scenarios are there in this category?

Reviewer_JRVpOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper targets on a practical problem: a trained LLM agent may lack knowledge about unfamiliar environment and hence performs badly. The proposed RSIAgent firstly let the agent to actively explore the environment and collect knowledge into its memory, then freeze the memory and do target tasks. Experiments show that RSIAgent effectively improves performance on specific agentic benchmarks. However, RSIAgent is more likely a test-time scaling method, and not fully aligned with the concept of RSI.

Critical strengths
  1. This paper has practical motivation: Acquiring environment-specific knowledge without updating model parameters is useful for agents operating in unfamiliar or private software environments.
  2. The orchestration of RSIAgent is sensible for collecting environment knowledge: a curriculum agent that drives the framework to continually explore, a verification agent that constructs useful memory, and an actor agent that interact with the environment.
Critical weaknesses
  1. The evaluation reuses target tasks during memory construction and does not establish held-out generalization. Both exploration stages are conditioned on the target query, and DRS explicitly attempts the target task and consolidates the resulting experience before final evaluation. Consequently, resetting the environment and freezing the memory do not make the final task unseen. The concern is that the experimental protocol cannot distinguish reusable environment adaptation from learning how to solve the particular evaluated instance. The paper needs a setting in which memory is constructed on exploration tasks and evaluated on separate, previously unseen target tasks.
  2. RSIAgent is more like a test-time scaling method, rather than RSI: The system repeatedly explores a specific task and finally solves it. The only difference is that the task may be modified by the curriculum agent. However, there is no comparison between other test-time scaling methods under the same computation budget.
Questions for authors
  1. How much improvement remains when exact target-task attempts are excluded from memory construction and evaluation uses unseen tasks?
  2. How does RSIAgent compare with other test-time scaling methods like repeated attempts, iterative refinement, and reflection-based memory under the same token budget?
  3. What's the exact meaning of RSIAgent (w/o RSI) in the experiments? Without RSI, RSIAgent is a plain actor agent with blank memory and the performance should be the same with the baselines. But in Section 4.2, it says that it already exceeds the open-source backbones. This is confusing.
6 comments

Dear Reviewer JRVp,

Thank you for your constructive review. We address query exposure and unseen-task evaluation, test-time scaling, and the w/o RSI baseline below.

1. “The evaluation reuses target tasks during memory construction and does not establish held-out generalization. Needs a setting evaluated on separate, previously unseen target tasks” (W1 & Q1)

We appreciate the reviewer’s thorough critique of our experiment design. We want to clarify that providing the test query to constrain the large search space of long-horizon tasks is a practical and well-established technique, which has precedents in transductive learning and test-time training [R1][R2]. Our system only uses test questions to conduct target-conditioned exploration and has no access to any gold answers or official scores. It lets the curriculum agent focus practice on the user’s goal, while the actor agent learns through verifier agent feedback.

We conducted a query-blind experiment to test adaptation before the evaluation query is revealed. BRS built fresh memory through broad exploration of the application environment. The curriculum agent then used this memory in DRS to propose a challenging but feasible, comprehensive practice task with verifiable subgoals. The actor agent practiced on it and updated memory using verifier agent feedback. All roles were blind to evaluation queries and target-specific artifacts. Memory was frozen and the environment reset before revealing the query for final execution, using the baseline’s backbone, harness, and execution cap. We will add the experiment and protocol details to the revision.

New experiment 1. Query-blind RSIAgent by domain (partial score, %).

Domain / task typeBaselineRSIAgent w/o queryRSIAgent w/ query
Spreadsheets5.0057.0268.69
Presentations25.0045.0050.00
Documents & email0.000.0060.00
Web interaction0.00100.00100.00
CAD design25.0057.4768.97
Electronic design38.4567.4185.25
Image editing60.0080.0080.00
Video editing50.5063.5091.00
Medical imaging36.0144.2358.36
Visual inspection0.0050.0050.00
Overall26.2956.0971.11

Overall binary accuracy: Baseline 0.00%; w/o query 0.00% on raw official scores (8.33% with the Web date diagnostic); w/ query 16.67%.

Results and analysis. On these tasks, the query-blind variant reaches 56.09% partial score versus 26.29% for the baseline (+29.80 pp), with gains in 9 of 10 domains. The improvement therefore persists when the evaluation query is withheld throughout memory construction.

The spreadsheet trajectory illustrates the reuse of application knowledge. Practice produced procedures for formulas, cached values, and native charts. At evaluation, the actor agent retrieved these notes, recalculated stale outputs, and checked dependent cells after changing an input. Across the spreadsheet domain, the partial score rises from 5.00% to 57.02%. A visible query further focuses this learning: target-conditioned exploration yields memory aligned with the task’s constraints (71.11% w/ query versus 56.09% w/o query overall).

2. “RSIAgent is more like a test-time scaling method, rather than RSI. No comparison between other test-time scaling methods.” (W2 & Q2)

We would like to point out that RSIAgent is an agent-level RSI within the broader test-time scaling framework [R3][R4][R5]. After each exploration round, memory is updated with lessons from verifier agent feedback. The curriculum agent uses these lessons to design the next practice tasks around remaining capability gaps, while the actor agent uses them to improve subsequent execution. To test whether the gains arise simply from more computation, we compare it with repeated attempts, iterative refinement [R6], and reflection memory [R7] using the same harness and RSIAgent-based token and tool-call limits. We thank the reviewer for the insightful feedback and we will add a more detailed discussion about the relationship between RSIAgent and other test-time scaling methods.

New experiment 2. Same-harness comparison with token and tool-call limits.

A. Selected task set · method-specific stopping

actor agentMethodPartial (%)Binary (%)Tokens (M)Tool callsCost (USD)
GLM-5.3Repeated attempts40.418.337932,311$364
GLM-5.3Iterative refinement [R6]43.240.004302,041$205
GLM-5.3Reflection memory [R7]53.0616.674572,334$206
GLM-5.3RSIAgent71.1116.672,90019,600$1,160

B. Five-task subset · method-specific stopping

actor agentMethodPartial (%)Binary (%)Tokens (M)Tool callsCost (USD)
GLM-5.3Repeated attempts42.0020.00193595$87
GLM-5.3Iterative refinement [R6]42.000.00100666$48
GLM-5.3Reflection memory [R7]48.0020.0093610$41

C. Same five-task subset · continue to resource limits

actor agentMethodPartial (%)Binary (%)Tokens (M)Tool callsCost (USD)
GLM-5.3Repeated attempts50.0020.001,0096,704$493
GLM-5.3Iterative refinement [R6]42.000.001,0157,134$494
GLM-5.3Reflection memory [R7]56.0020.001,0007,176$512
GLM-5.3RSIAgent76.0040.001,0597,165$423

Experimental setup. We use the OSWorld 2.0 tasks from the query-blind experiment. All methods see the target query and share the GLM-5.3 actor agent, execution harness, and Kimi-K3 verifier agent. Token and tool-call limits follow RSIAgent’s measured usage on the five tasks with logs and the full-set mean on the remaining tasks. Official scores never guide stopping or candidate selection; usage includes all agent roles and discarded attempts. Baseline costs are provider-billed; RSIAgent costs are estimated assuming a 97% cache-hit rate. Both exclude the official evaluator and user simulator. We will add the full experimental details to the revision.

Results and analysis. The same-harness w/o RSI baseline scores 26.29%. With their own stopping rules, TTS baselines reach 40.41–53.06%, versus 71.11% for RSIAgent (+18.05 pp over the strongest baseline). To test whether early stopping explains the gap, we also continue the baselines to the resource limits on the five-task subset. They consume 1,000–1,015M tokens, close to RSIAgent’s 1,059M, but reach at most 56.00% versus 76.00% (+20.00 pp). The gap persists when additional computation is used, supporting the value of structured exploration and accumulated environment knowledge.

3. “What exactly is RSIAgent (w/o RSI)?” (Q3)

RSIAgent (w/o RSI) retains our code-as-policy actor agent–verifier agent harness, with empty persistent memory and no exploration stage (§4.1; Appendix C.2). Its advantage over a raw language model is a harness gain. Comparing it with full RSIAgent isolates the added exploration and memory stage. We will clarify these labels in §4.2 of the revision.

References

[R1] Palavalli et al. (2024). A Taxonomy for Data Contamination in Large Language Models. CONDA, §2.2.3.

[R2] Sun et al. (2020). Test-Time Training with Self-Supervision for Generalization under Distribution Shifts. ICML.

[R3] Snell et al. (2025). Scaling LLM Test-Time Compute Optimally Can be More Effective than Scaling Parameters for Reasoning. ICLR.

[R4] Muennighoff et al. (2025). s1: Simple Test-Time Scaling.

[R5] Shen et al. (2025). Thinking vs. Doing: Improving Agent Reasoning by Scaling Test-Time Interaction. NeurIPS.

[R6] Madaan et al. (2023). Self-Refine: Iterative Refinement with Self-Feedback. NeurIPS.

[R7] Shinn et al. (2023). Reflexion: Language Agents with Verbal Reinforcement Learning. NeurIPS.

Thanks for the response from authors. Most of my concerns have been fixed, but I'm still confused about Q3. You said that RSIAgent (w/o RSI) retains the code-as-policy agent–verifier agent harness, with empty persistent memory and no exploration stage, while Appendix C.2 says the baseline uses the same code-as-policy actor-verifier harness. So what's the exact difference between the baseline and RSIAgent (w/o RSI) in Table. 2? Without persistent memory and exploration stage, the verifier has no effect on the behavior of the actor and is useless, in such case, what's the difference between your harness and a basic ReAct agent?

Reviewer_BtuWOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents RSIAgent, a training-free multi-agent framework that coordinates curriculum, actor, and verifier agents to construct memory through broad exploration and target-conditioned practice. Empirical results show that RSIAgent achieves better performance on open-source models compared with SOTA models.

Critical strengths
  1. The paper presents a coherent framework integrating autonomous exploration, execution-grounded verification, and persistent memory, providing a concrete approach to agent adaptation without parameter updates.
  2. Experiments on different benchmarks show the potential of RSIAgent.
Critical weaknesses
  1. In the experimental setting, the whole system has already known the information about the testing questions. The DRS also had some trials on the questions. While in the paper, it was claimed that the agent only explores on different tasks, which might be different from the experimental setting. Also this could conduct an unfair comparison with other models.
  2. The main contribution to the performance improvements mainly come from the RSIAgent framework, rather than the RSI method.
Questions for authors
  1. If split the dataset into two sets by the task, only train on one set and test on another one. What is the performance?
6 comments

Dear Reviewer BtuW,

Thank you for your careful review and for recognizing the framework’s coherent design. We address your concerns about query exposure and the contribution of the RSI stage below.

1. “the whole system has already known the information about the testing questions. Paper claims agent only explore on different tasks” (W1)

Conditioning adaptation on the task input has precedents in transductive learning [R1] and test-time training [R2]. Reflexion, Self-Refine, and EvoTest likewise use task-conditioned feedback or repeated interaction, with different feedback signals [R3][R4][R5]. Our method uses the query to constrain the vast search space of long-horizon tasks, without access to gold answers or official scores. The verifier agent evaluates outcomes and intermediate attempts from observable evidence, and the actor agent uses this feedback to improve its behavior and memory.

The paper does not claim that exploration is restricted to tasks different from the evaluation target. Section 3.2 (p. 5) describes DRS beginning with a target attempt, and §4.1 (p. 6) states that both exploration stages use the target query. The restriction against attempting the unchanged target applies only to BRS (Appendix B.2). We will clarify this distinction in the revision.

2. “main contribution to the performance improvements mainly come from the RSIAgent framework, rather than the RSI method” (W2)

Table 1 already isolates the additional gains from the RSI stage while holding the backbone and actor agent–verifier agent harness fixed. Adding exploration and memory improves the partial score on all three benchmark splits: OSWorld 2.0, ALE Near-term, and ALE Last-Exam.

Existing evidence. Same-harness ablation from main-paper Table 1.

BenchmarkPartial (%) without → with RSIΔ (pp)Binary (%) without → with RSIΔ (pp)
OSWorld 2.071.97 → 78.98+7.0137.80 → 42.68+4.88
ALE Near-term83.75 → 85.53+1.7849.25 → 53.73+4.48
ALE Last-Exam41.19 → 41.91+0.7216.67 → 16.670.00

3. “only train on one set and test on another one. What is the performance?” (Q1)

To address your task-split question, we conducted a query-blind experiment. BRS built fresh memory through broad exploration of the application environment. The curriculum agent then used this memory in DRS to propose a challenging but feasible, comprehensive practice task with verifiable subgoals. The actor agent practiced on it and updated memory using verifier agent feedback. All roles were blind to evaluation queries and target-specific artifacts. Memory was frozen and the environment reset before revealing the query for final execution, using the baseline’s backbone, harness, and execution cap. We will add the experiment and protocol details to the revision.

New experiment 1. Query-blind RSIAgent by domain (partial score, %).

Domain / task typew/o RSI (same harness)RSIAgent w/o queryRSIAgent w/ query
Spreadsheets5.0057.0268.69
Presentations25.0045.0050.00
Documents & email0.000.0060.00
Web interaction0.00100.00100.00
CAD design25.0057.4768.97
Electronic design38.4567.4185.25
Image editing60.0080.0080.00
Video editing50.5063.5091.00
Medical imaging36.0144.2358.36
Visual inspection0.0050.0050.00
Overall26.2956.0971.11

Overall binary accuracy: Baseline 0.00%; w/o query 0.00% on raw official scores (8.33% with the Web date diagnostic); w/ query 16.67%.

Results and analysis. On these tasks, query-blind RSIAgent reaches 56.09% partial score, compared with 26.29% for the baseline (+29.80 pp). It improves 9 of the 10 domains and ties the remaining domain. The framework retains a substantial benefit when the curriculum agent supplies the practice goal itself.

The spreadsheet results illustrate how environment knowledge transfers to an unseen query: the domain partial score rises from 5.00% to 57.02%. In a representative trajectory, practice produced notes on formulas, cached values, and native charts. At evaluation, the actor agent retrieved these notes, recalculated stale outputs, and checked that dependent cells changed with the input.

With the query available, RSIAgent can further direct this learning toward the final task and achieve stronger performance.

References

[R1] Palavalli et al. (2024). A Taxonomy for Data Contamination in Large Language Models. CONDA, §2.2.3.

[R2] Sun et al. (2020). Test-Time Training with Self-Supervision for Generalization under Distribution Shifts. ICML.

[R3] Shinn et al. (2023). Reflexion: Language Agents with Verbal Reinforcement Learning. NeurIPS.

[R4] Madaan et al. (2023). Self-Refine: Iterative Refinement with Self-Feedback. NeurIPS.

[R5] He et al. (2026). EvoTest: Evolutionary Test-Time Learning for Self-Improving Agentic Systems. ICLR.

Dear Reviewer BtuW,

Thank you again for your review. We would appreciate your feedback on whether our responses and additional experiments address your concerns. For convenience, we summarize the evidence below, including an additional comparison relevant to W2.

W1 (query exposure and fairness). Conditioning exploration on the target query is explicitly described in §3.2 and §4.1. The restriction against attempting the unchanged target applies specifically to BRS (Appendix B.2). The query guides exploration through the large search space of long-horizon tasks, without providing reference answers or official scores. All methods in the additional compute comparison below also receive the target query.

W2 (framework vs. RSI stage). Table 1 already isolates the contribution of exploration and memory while holding the backbone and actor agent–verifier agent harness fixed, showing partial-score gains on all three benchmark splits. The query-blind experiment below provides further evidence using the same execution harness. We additionally compared RSIAgent with repeated attempts, Self-Refine-style iterative refinement, and Reflexion-style memory using the same actor agent, verifier agent, and execution harness. On the same five-task subset, the methods use comparable resources: 1.000–1.059B tokens and 6,704–7,176 tool calls. RSIAgent reaches 76.0% partial score versus at most 56.0% for these alternatives. The full protocol and results are available in the discussion.

Q1 (evaluation on unseen target queries). We conducted a query-blind experiment: BRS and DRS construct memory through autonomous exploration and self-generated practice, with all agents blind to evaluation queries and target-specific artifacts. Memory is frozen and the environment reset before the evaluation query is revealed. On this task set, the partial score increases from 26.29% to 56.09% over the same-harness baseline, improving 9 of 10 domains and tying the remaining one. The protocol and domain-level results are included in our earlier response.

If any concerns remain unresolved, could you identify the specific points? We would be glad to clarify them and would appreciate your reassessment in light of these clarifications and additional results.

Best regards,

The Authors

Comments on the paper

1 comment

Dear Program Chairs, Senior Area Chairs, Area Chair, and Reviewers,

Thank you for your time and constructive feedback. As the discussion draws to a close, we briefly summarize its outcomes. The additional experiments and clarifications are now incorporated into the revised manuscript, with additions highlighted in blue.

  • Reviewer JRVp recognized the practical motivation and agent design. To address the concern about unseen-task evaluation, we added query-free exploration, which raises partial score from 26.29% to 56.09%, demonstrating gains from memory built without target queries (§4.3; Table 3). To test whether the gains simply reflect additional computation, we compared TTS methods using the same execution harness and comparable resources. RSIAgent achieves 76% versus at most 56% (Appendix A.2). To resolve the baseline ambiguity, we clarify that w/o RSI retains verifier agent feedback but removes exploration and persistent memory (§4.2; Appendix C). JRVp confirmed that the remaining ambiguity was resolved and raised the rating to 3.

  • Reviewer BMx3 praised the system design, exploration strategy, and RSI ablations. In response to the concern about the RSI claim, we clarify that RSIAgent implements agent-level RSI within test-time scaling: verifier agent feedback guides memory updates, which shape the curriculum agent’s subsequent practice tasks and the actor agent’s execution (§3.2). Learning in one round thus changes how the agent explores and improves in the next, with performance gains across rounds shown in Figure 3. The requested experiments show gains from query-free exploration and better performance than other TTS methods under comparable resource consumption (§4.3; Appendix A.2). For the follow-up question about memory differences, we identify missing document-reconstruction and heading-preservation procedures in memory built through query-free exploration (Appendix D.8). BMx3 confirmed that all concerns had been addressed and maintained a positive assessment.

  • Reviewer BtuW recognized the coherent way RSIAgent combines autonomous exploration, verification, and persistent memory. To address the concern about target access, we showed that query-free exploration improves partial credit in nine of the ten OSWorld domains tested (§4.3, Table 3). To measure what RSI contributes, we compared the same execution harness with and without RSI across all 82 OSWorld tasks. Adding RSI raises mean partial credit from 71.97% to 78.98% (Table 1). Our follow-up clarifies that the closed-source comparison uses each system’s own harness and budget (Appendix B.3). It also shows that both frozen-memory runs outscored the no-RSI baseline on each of the four ablation tasks in partial credit, with binary credit staying at zero in both conditions (Figure 4, Appendix B.4). BtuW acknowledged that the paper describes target practice during DRS, welcomed the query-free experiment, and raised the rating.

We sincerely appreciate the discussion, which has helped us strengthen both the evidence and presentation.

Best regards,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.