acceptodds
Under review as a conference paper at ICLR 2027

Learn Now, Use Next, Trust Later: Prequential Test-Time Learning for LLM Agents

Abstract

Adapting large language model agents during deployment requires not only retaining past experience, but also turning new observations into timely guidance. Many test-time learning methods, however, acquire knowledge from completed episodes, even when memory is retrieved at every step. Feedback from an ongoing interaction may therefore not be distilled into knowledge soon enough to help the next decision. Acquiring knowledge at the granularity of individual transitions could reduce this delay, but raises a separate challenge: a rule that is useful within one episode may not be reliable enough to guide future episodes. Waiting for validation can forfeit immediate benefits, whereas unrestricted reuse can propagate accidental or misattributed guidance. We introduce StepLearn, a nonparametric framework that separates immediate use from persistent trust. It turns informative transitions into hypotheses that can guide the next step, while requiring prospective validation before reuse across episodes. Its predicted effects are checked against subsequent observations outside the source episodes, and only sufficiently supported hypotheses become available for persistent guidance. This process updates external knowledge while keeping all model parameters fixed. Over five rounds on WebArena-Lite and ALFWorld, StepLearn achieves average success rates of 59.9% and 84.0% with GPT-5-mini, and 57.8% and 88.1% with Qwen3.5-35B-A3B, respectively. It outperforms EvoTest, the strongest baseline, by 2.2–12.7 percentage points across the four settings. Learning dynamics further show that these gains are not restricted to the final repetition, with advantages already present on first task attempts in most settings.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Large language model agents, Test-time learning, Prequential learning, Agent memory, Online adaptation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_8CQgOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes StepLearn, a test-time learning framework that allows for step-level learning within a task episode. Lessons learned during an episode can be used immediately, but need further validation before they can be reused in later episodes. Each new lesson is stored as a hypothesis with its own matching and validation rules, and is automatically checked against experience from later episodes. Once validated, the hypothesis is promoted into a persistent memory that can be retrieved in future episodes, whereas hypotheses with enough counterevidence are removed from retrieval.

The main contribution is a new perspective that TTL can be performed at the step level, together with a step-level learning method combined with a hypothesis validation framework, while most existing TTL methods only learn after an episode ends. Experiments on WebArena-Lite and ALFWorld show that StepLearn achieves the highest average success rate over existing baselines in all four settings.

Critical strengths

The problem is clear and well motivated. The paper highlights an important question of step-level learning, while existing TTL methods mostly operate at the episode level. The problem formulation is clean.

The experiments cover a broad set of baselines, including recent TTL methods and parameter-training methods (SFT and TTRL), with both an API and a locally deployed actor. The cost analysis is also helpful, showing that the auxiliary learner accounts for only a small fraction of the total cost.

Critical weaknesses

The source of the gains does not seem very clear.

The core claim of the paper is that step-level learning leads to better test-time learning. While I understand the intuition that a lesson learned at step t can help fix a mistake within the same episode, I do not see this clearly in the current experiments. Removing runtime reuse reduces performance by only about 1–3 points in three of the four settings, and I could not find a systematic analysis of how often a runtime hypothesis is used and then leads to success within the same episode.

At the same time, in three of the four settings, StepLearn’s advantage is already present in Round 1 (Figure 4), and Figure 3 shows limited improvement across rounds in most settings. It is therefore unclear to me how much of the gain comes from step-level learning, and how much from knowledge accumulated on earlier tasks. Related to this, I am also not entirely sure what the authors intend to claim as the main evidence of effective test-time learning: is it sufficient that StepLearn outperforms the baselines in Round 1, or should it also show improvement across rounds on the same task?

To help separate the sources of the gains, could the authors provide some additional results? For example: 1. running each task’s 5 episodes independently (without sharing memory across tasks) for both StepLearn and the baselines; 2. disabling StepLearn’s persistent memory so that only step-level learning (within an episode) remains.

The reliability of the validation mechanism.

The “trust later” component relies on matching hypotheses to actions, but as far as I understand, candidates are matched only by scope and action type (§3.4), while the natural-language condition and policy are not used. If so, a hypothesis might be promoted or downgraded based on an action that does not follow its policy, e.g., clicking a different element on the same page type. In addition, the promotion thresholds (ρ=0.67\rho = 0.67, m=2m = 2) are described as “a heuristic evidence gate” (Appendix A.3), and it is not entirely clear to me how these numbers were chosen. Since prequential validation is a key part of the contribution, it would be helpful to better understand how reliable it is in practice.

Questions for authors
  1. Could the authors clarify how “scope” and “action type” are used for matching, and how accurate the promotion mechanism is in practice? For example, how often are promoted hypotheses validated by actions that are actually consistent with their policy?

  2. Under the current experiment setup in the paper, a round consists of a stream of different tasks, and the stream is repeated for 5 rounds. What do the authors view as the main sign of effective test-time learning: learning across tasks within a round, or learning across rounds on the same task?

  3. The central claim of the paper is to perform step-level learning at test-time rather than doing it only at episode-level, and the prequential validation seems designed to make step-level knowledge safe to reuse across episodes, rather than to improve cross-episode learning itself. Could the authors clarify whether StepLearn is also expected to be better at cross-episode learning? If so, could the authors explain in simple terms, why is StepLearn better at cross-episode learning compared to existing baselines (how does the step-level learning mechanisms translate to better cross-episode learning)?

  4. StepLearn’s setup differs from existing TTL methods such as EvoTest and JitRL, which run all episodes of one task consecutively before moving to the next task. Could the authors clarify how these baselines were adapted to the round-based protocol to ensure a fair comparison?

Reviewer_5CWYOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies test-time learning for LLM agents with frozen parameters, sparse binary rewards, and recurring tasks, where improvement comes only from an external knowledge store. It argues that existing methods learn only after an episode ends, so feedback from the current transition cannot inform the next step, and that learning from single transitions risks propagating unreliable rules. Both problems are motivated conceptually, through a hypothetical example and a schematic figure.

StepLearn turns informative transitions into hypotheses (a condition, a policy, and observable predicted effects). A runtime copy can guide the next step within the current episode, while a hidden candidate copy must have its frozen predictions confirmed in later, non-source episodes before entering verified memory, which is further pruned by retention safeguards.

Experiments use only two mid-sized actors (GPT-5-mini and Qwen3.5-35B-A3B) on WebArena-Lite and ALFWorld, with each task repeated for five rounds. StepLearn reports 59.9/84.0% and 57.8/88.1% average success, 2.2 to 12.7 points above EvoTest, and claims to outperform parameter training, with SFT and TTRL reaching 58.5% and 73.5% on ALFWorld. Supporting evidence consists of aggregate success rates, first-attempt and per-round curves, and component ablations.

Critical strengths

S1. A clear, auditable separation between using and trusting new knowledge. A new rule can guide the next step immediately, but enters persistent memory only after its frozen predictions are confirmed by deterministic checks in non-source episodes. This replaces the LLM self-judgment or voting that most memory methods rely on, which makes it a useful design principle for addressing memory pollution, even though the paper does not show that this gate drives its gains.

S2. Consistent gains at low cost. StepLearn is best in all four settings and costs less than EvoTest, with learner calls under 6% of the total and small learners matching larger ones. This practicality is the main empirical case for acceptance, but its weight depends on whether the gains come from the proposed mechanisms rather than the harness.

Critical weaknesses

W1. The existence of the targeted problems, and the method’s effect on them, are not demonstrated. Both problems are motivated only by a hypothetical example. The paper does not show that baseline agents repeat mistakes within an episode after informative feedback, although this feedback is already in the actor’s context and may be lost only because the harness compresses past observations. Nor does it evaluate a variant without validation, so the harm of unvalidated rules is not shown. The method is supported only by aggregate results, which are mixed: removing runtime reuse costs only 0.9–2.2 points on ALFWorld, the largest ALFWorld drop comes from the retention safeguards rather than the validation gate, and the only case study does not show within-episode use. Since the paper’s contributions are mechanistic, this gap is central.

W2. The positioning is unclear, and the experiments do not support either plausible framing. If StepLearn is a way to make cheaper models competitive, it should be compared with stronger models used directly; if it is a general framework, it should be shown to help strong models, which may already self-correct from context. Neither is done, as only two mid-sized actors are evaluated. The novelty is also unclear relative to prior rule-learning methods such as ExpeL and AutoManual; the latter, which updates and checks rules online without training, reports 86.2% on ALFWorld with GPT-3.5-turbo but is not cited. Finally, the claimed advantage over parameter training rests on weak baselines: SFT is trained for one epoch on the base model’s own trajectories, including failures, and TTRL is initialized from it and evaluated for only two rounds, whereas RL-trained 1.5B–7B models often exceed 90% on ALFWorld, albeit with training data. It therefore remains unclear what StepLearn adds beyond existing methods.

Questions for authors

Clarification questions

Q1. The ALFWorld SFT baseline is trained for one epoch on the base model’s own trajectories, including unsuccessful ones, and TTRL is initialized from it and evaluated for two rounds instead of five. What motivated these choices, and how does StepLearn compare with an SFT or RL baseline trained on expert trajectories or the ALFWorld training split?

Q2. Is StepLearn intended as a way to make cheaper models competitive, or as a general test-time learning framework? In the former case, how does it compare with stronger models used directly without learning?

Requested analyses that could change my assessment

Q3. In baseline failure trajectories, what fraction of failures involve repeating the same error or loop within an episode after informative feedback was observed? Does this problem persist when the actor receives the full, uncompressed interaction history?

Q4. Can the authors evaluate a variant that promotes every hypothesis immediately without validation, to show that unvalidated rules are actually harmful?

Q5. Which tasks require within-episode learning (e.g., episodes with acquisition triggers that are later recovered), and is the drop from removing runtime reuse concentrated on them?

Reviewer_Nb99OpenReview ↗
Rating
3(+1)
Confidence
2(−2)
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

StepLearn enables LLM agents to learn from individual interaction steps by extracting testable policy hypotheses that can guide the next decision within the current episode. It separates immediate use from persistent trust by requiring validation against subsequent observations outside the source episodes before allowing cross-episode reuse, while keeping all model parameters fixed.

Critical strengths
  • The paper clearly explains why learning should be both timely and reliable.
  • The figures make the main idea easy to understand.
  • New rules help immediately.
  • The method shows gains across two environments and two model families.
Critical weaknesses

My main concerns are the limited experimental evidence and, more importantly, the novelty of the algorithm itself.

(1) The validation step needs stronger evidence. The ablations in Table 2 test step-level learning, immediate use, and the removal of poor rules. They do not directly test the value of validation before long-term reuse. Comparing no validation, model confidence, and validation in later episodes would help. A separate test with frozen memory on new tasks would also provide stronger evidence of transfer. The paper should report rule error rates and show how different validation thresholds affect rule quality and learning speed.

(2) The method lacks sufficient technical novelty. SCOPE (Pei et al., 2025) already learns guidance during task execution, uses it in subsequent steps, and separates temporary guidance from long-term memory. These ideas cover much of StepLearn’s core design. StepLearn adds checks based on later observations outside the source episodes and keeps candidate rules hidden during validation. I view these changes as a refinement of the existing memory design, rather than a substantial new learning method.

Questions for authors

See the weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.