acceptodds
Under review as a conference paper at ICLR 2027

PILOT in the Loop: Live Self-Improvement for Long-Horizon Agents

Abstract

Long-horizon agent runs generate experience that can improve both the current run and future runs: successful attempts reveal reusable procedures, while failed attempts expose failure modes. Most self-improvement methods process this experience only after a run ends, so they can neither recover that run nor immediately apply and validate the lessons learned, making self-improvement less efficient and less reliable. We argue that self-improvement should instead be live, using emerging experience both to redirect the active run and to update the persistent harness. However, live self-improvement exposes an architectural gap: existing agent architectures do not simultaneously support live correction and a dedicated self-improvement role. Single-agent self-correction can revise the active run, but the same agent must execute the task and judge its trajectory within a limited context, splitting attention between execution and oversight. Subagent delegation separates execution from the main agent, but the main agent typically cannot redirect the active subagent while the subagent is still running. To this end, we present PILOT, a supervisor–worker harness that realizes live self-improvement through two coupled mechanisms: (1) live steering lets a separate supervisor redirect or abort the active worker during execution; and (2) live self-evolution distils procedures and failure modes revealed during execution into reusable skills and memory. Across three frozen backbones and four benchmarks, PILOT ranks first in eight of twelve configurations. On Terminal-Bench 2.0, PILOT outperforms counterpart harnesses by as much as 11.8 percentage points. In the self-improvement setting, PILOT gains 14.6 points with GLM-5.1; mean output tokens fall by 42.9%, and successful evaluations per million output tokens rise by 110.3%.

Primary area:
foundation or frontier models, including LLMs
Keywords:
agent, LLM, Long-Horizon

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_s2obOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents PILOT, a supervisor–worker framework for live self-improvement in long-horizon agents. By integrating live steering with live self-evolution, PILOT enables supervisors to correct workers during task execution while continuously distilling reusable skills and memory from execution experience. This design improves both ongoing task performance and future task efficiency without modifying the underlying model parameters. Experiments across multiple models and benchmarks demonstrate its effectiveness in increasing task success rates, reducing token consumption, and transferring accumulated experience across tasks.

Critical strengths

Strong Motivation. The paper addresses an important limitation of existing agent self-improvement approaches, which primarily rely on post-hoc reflection and cannot benefit the ongoing execution. The idea of enabling real-time steering and continuous improvement during inference is well motivated, particularly for long-horizon tasks where errors can accumulate and early intervention can prevent execution failures.

Critical weaknesses

1. Insufficient evidence for the effectiveness of the proposed live steering mechanism. PILOT introduces an additional supervisor that shares the same backbone model as the worker; however, the paper lacks sufficient ablation studies and compute-matched comparisons to isolate the benefits of live steering. It therefore remains unclear whether the performance gains stem from the proposed mechanism itself or merely from additional inference compute and context separation. This weakens the evidence supporting the paper's central methodological contribution.

2. Limited evidence for reliable and generalizable self-improvement. The self-improvement experiments repeatedly evaluate the same set of tasks while accumulating task-specific skills, raising concerns that the observed improvements primarily reflect experience reuse on previously encountered tasks rather than generalizable capability improvement. Moreover, the framework relies on task-level verifier outcomes to retain harness updates, without validating the correctness or usefulness of individual skills. Although cross-benchmark transfer is evaluated, the evidence remains limited. These issues undermine the paper's broader claims of effective and reliable self-improvement.

Questions for authors
  1. What are the primary sources of PILOT's performance gains? Given that the supervisor and worker share the same backbone model, it remains unclear whether the improvements arise from live steering, context separation, or simply additional inference compute. Could the authors provide more comprehensive ablation studies and compute-matched comparisons to disentangle these factors and better isolate the contribution of the proposed mechanisms?
  2. To what extent does PILOT achieve generalizable self-improvement? The self-improvement experiments repeatedly evaluate the same set of tasks while accumulating task-specific skills, raising concerns that the observed gains may primarily reflect memorization or reuse of task-specific solutions rather than generalizable improvements. Could the authors evaluate PILOT on a sequence of previously unseen tasks to provide stronger evidence of its ability to continuously improve and generalize?
  3. How are the correctness and effectiveness of steering decisions and harness updates validated? The supervisor generates steering instructions without access to ground-truth feedback, while accumulated skills are retained based on task-level success rather than their individual validity. Could the authors provide a more systematic analysis of ineffective or harmful steering interventions, as well as the correctness and utility of the accumulated skills?
  4. Are the paper's central claims regarding "self-improvement" and "long-horizon agents" sufficiently substantiated? The proposed framework largely relies on supervisor-guided inference and persistent experience accumulation, making its distinction from existing multi-agent supervision and memory-augmented approaches somewhat unclear. Furthermore, the proposed mechanisms are not inherently specific to long-horizon tasks, and the experiments do not explicitly establish how performance gains depend on task horizon length. Could the authors clarify the conceptual novelty of the proposed self-improvement framework and provide stronger evidence supporting its particular relevance to long-horizon tasks?
Reviewer_U9q9OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces PILOT, a supervisor–worker harness for long-horizon agents. A supervisor monitors workers in separate contexts, redirects or aborts their execution, and records reusable skills and memory without updating model weights. The evaluation covers three backbones and four benchmarks, with particularly strong results on SWE-bench Pro. Repeated evaluation on Terminal-Bench also shows improvements in best-observed success rates and output-token efficiency. The approach is practically relevant, but the experiments do not adequately separate the benefits of live intervention from additional inference, task-specific experience, and verifier-based update selection. The results are promising, while the central explanation for those results remains insufficiently established.

Critical strengths
  1. Separating execution from oversight is a sensible design for long-horizon tasks. The notification, question, steering, and abort interfaces make the interaction concrete, and the examples demonstrate how a supervisor can identify an error or redirect an unproductive strategy before execution finishes.

  2. The experiments include multiple backbones and task domains. PILOT performs particularly well on SWE-bench Pro, leading with all three models and achieving an average score of 67.9 compared with 63.0 for Pi. These results provide encouraging evidence that the harness can improve practical software engineering performance.

  3. The paper clearly describes when experience is generated and retained. Tasks within an iteration receive isolated copies of the shared harness, and verifier outcomes are unavailable during execution. Counting both supervisor and worker output tokens also makes the reported generation-efficiency measurements more informative.

Critical weaknesses
  1. The evaluation does not isolate the contribution of live steering. Compared with the single-agent baselines, PILOT changes the number of reasoning contexts, delegation structure, available inference, and communication mechanism simultaneously. A comparison against the same supervisor–worker architecture without live redirection is missing. Table 2 and the trajectory examples show plausible instances of useful intervention, but they do not establish how much of the overall performance gain comes from steering or how often supervision introduces errors.

  2. The self-improvement protocol primarily measures adaptation to recurring benchmark tasks. Each iteration revisits the same 89 Terminal-Bench tasks, and Appendix A.3 explicitly asks agents to store one skill named after each task, including the exact tools, commands, and setup used. This is a useful recurring-task setting, but it does not distinguish general procedural learning from retrieving task-specific solutions. Retaining updates only from successful runs further introduces outcome-dependent selection. Its contribution is not separated from that of generating updates during live execution.

  3. The strongest improvement claims rely on best-so-far results across 20 iterations. Such curves cannot show whether the evolving harness improves consistently, fluctuates substantially, or regresses after reaching its maximum. Selecting the best evaluation also incorporates favorable run variation into the reported gain. Although the protocol reports two runs per configuration, the paper does not provide uncertainty estimates or enough detail about independent adaptation histories. The reported maxima and narrow benchmark advantages should therefore be interpreted cautiously.

  4. The paper’s architectural positioning is broader than the evidence supports. Section 5 acknowledges systems that coordinate agents during execution and Continual Harness, which adapts harness components within a reset-free run. The remaining distinction needs to be stated more precisely: asynchronous communication, intervention during an active worker session, selective trajectory inspection, or the coupling of intervention with persistent updates. Without direct comparisons or targeted ablations, the paper does not establish which of these properties provides a substantive advantage over existing supervision and adaptation patterns.

  5. Experience transfer is uneven and does not yet support a general claim of reusable improvement. In Figure 3, transferred experience improves GLM-5.1 from 26.5 to 30.8 or 35.1. For Kimi-K2.6, however, GLM-generated experience changes the score only from 33.8 to 33.9, while its own experience reduces the score to 31.6. Fresh PILOT also trails Pi on GLM-5.1 LHTB. These results reveal meaningful dependence on the receiving backbone and experience source, but the paper offers little analysis of when skills help, fail to transfer, or become harmful.

Questions for authors
  1. Compare PILOT with a matched supervisor–worker baseline using identical prompts, tools, initial experience, worker limits, and inference budgets. Disable live redirection in one condition and replace it with synchronous checkpoints in another. This would test the value of the communication mechanism directly.

  2. Compare live experience extraction with post-hoc extraction under the same retention policy. How often does an update actually benefit another worker within the episode that produced it? Since workers load the harness when spawned, this measurement would clarify the practical importance of writing updates before the episode ends.

Reviewer_H7F2OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes PILOT, a supervisor-worker agent harness for live self-improvement in long-horizon tasks. The central idea is to separate task execution from supervision while allowing the supervisor to intervene during an ongoing worker execution. PILOT introduces two coupled mechanisms: live steering, which enables the supervisor to redirect or abort active workers, and live self-evolution, which extracts reusable skills and memory from execution experience to improve subsequent runs. The authors evaluate PILOT across three frozen backbone models and four long-horizon benchmarks, including Terminal-Bench 2.0, SWE-bench Multilingual, SWE-bench Pro, and LHTB. In the one-shot setting, PILOT achieves the highest performance in eight of twelve model-benchmark configurations. The paper further investigates self-improvement over 20 iterations on Terminal-Bench 2.0, reporting up to 14.6 percentage points of improvement in best-observed pass rate and substantial reductions in output-token consumption. Additional experiments examine live-steering behavior, accumulated skills, and cross-benchmark experience transfer.

Critical strengths

S1. I find the motivation convincing. Existing agent systems often rely on post-hoc reflection or self-improvement after task completion, which limits their ability to recover during ongoing execution. PILOT addresses this limitation through a supervisor-worker architecture that allows a separate supervisor to inspect execution progress and intervene when necessary. The proposed combination of live steering and persistent experience accumulation is a reasonable design for long-horizon agents. I particularly appreciate the separation of execution and supervision contexts, which allows the worker to focus on task execution while the supervisor monitors progress and identifies potential problems. The concrete live-steering examples also help illustrate how the proposed mechanism can correct unsuccessful strategies during execution.

S2. The paper evaluates PILOT across three backbone models and four challenging benchmarks, which provides useful evidence of its applicability across different agent capabilities and task domains. The improvements on Terminal-Bench 2.0 and SWE-bench Pro are encouraging, and I appreciate that the authors compare against several established agent harnesses while keeping the backbone fixed. The additional experiments on iterative self-improvement and experience transfer are also valuable. In particular, the observed improvements across iterations and the reduction in output-token consumption suggest that accumulated experience can help agents perform repeated tasks more efficiently.

Critical weaknesses

PILOT introduces several changes simultaneously, including a separate supervisor, live communication, worker redirection, abort and restart capabilities, and persistent skill updates. Although the paper provides comparisons against existing agent harnesses, I could not find controlled ablations that independently evaluate these components. For example, it would be important to compare the full PILOT framework against variants without live steering, without persistent self-evolution, or with the same supervisor-worker structure but only post-hoc experience updates. These comparisons would help establish whether the improvements come specifically from the proposed live mechanisms or from the additional supervision and orchestration resources.

I also have concerns about computational fairness. PILOT uses both supervisor and worker model calls and may launch multiple workers, whereas several baselines use a single-agent execution architecture. Holding the backbone and wall-clock time limit fixed does not necessarily provide an equivalent inference budget. Although the paper reports output-token efficiency across self-improvement iterations, I could not find a comprehensive comparison of total inference costs across the competing harnesses in the one-shot setting.

Questions for authors

Q1. Could the authors provide a more controlled ablation of the two proposed mechanisms? In particular, how does PILOT perform when live steering is disabled, when persistent self-evolution is disabled, or when the supervisor performs only post-hoc updates after a worker finishes? These comparisons would help determine the individual contributions of live supervision and live experience accumulation.

Q2. Could you clarify how the iterative self-improvement results generalize beyond the repeatedly evaluated Terminal-Bench 2.0 tasks? Since the evolution protocol creates task-specific skills and uses verifier outcomes to select retained updates, I would appreciate an evaluation on held-out tasks that were not encountered during evolution. Additionally, could the authors report the raw per-iteration pass rates, rather than only the best-so-far results, to better characterize improvement stability?

Q3. Could you provide a more detailed computational cost comparison between PILOT and the baseline harnesses? In particular, it would be helpful to report total input and output tokens, supervisor and worker calls, the number of worker attempts, and wall-clock execution time under comparable budgets. This would help determine whether the performance improvements are attributable to the proposed architecture rather than additional inference resources.

Q4. It would be better to discuss the mixed results of cross-benchmark experience transfer in Figure 3. For example, accumulated experience improves GLM-5.1 on LHTB but provides little or negative benefit for Kimi-K2.6. What factors determine whether accumulated skills transfer successfully to a different benchmark or backbone, and how are potentially irrelevant skills handled?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.