- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes StepLearn, a test-time learning framework that allows for step-level learning within a task episode. Lessons learned during an episode can be used immediately, but need further validation before they can be reused in later episodes. Each new lesson is stored as a hypothesis with its own matching and validation rules, and is automatically checked against experience from later episodes. Once validated, the hypothesis is promoted into a persistent memory that can be retrieved in future episodes, whereas hypotheses with enough counterevidence are removed from retrieval.
The main contribution is a new perspective that TTL can be performed at the step level, together with a step-level learning method combined with a hypothesis validation framework, while most existing TTL methods only learn after an episode ends. Experiments on WebArena-Lite and ALFWorld show that StepLearn achieves the highest average success rate over existing baselines in all four settings.
Critical strengths
The problem is clear and well motivated. The paper highlights an important question of step-level learning, while existing TTL methods mostly operate at the episode level. The problem formulation is clean.
The experiments cover a broad set of baselines, including recent TTL methods and parameter-training methods (SFT and TTRL), with both an API and a locally deployed actor. The cost analysis is also helpful, showing that the auxiliary learner accounts for only a small fraction of the total cost.
Critical weaknesses
The source of the gains does not seem very clear.
The core claim of the paper is that step-level learning leads to better test-time learning. While I understand the intuition that a lesson learned at step t can help fix a mistake within the same episode, I do not see this clearly in the current experiments. Removing runtime reuse reduces performance by only about 1–3 points in three of the four settings, and I could not find a systematic analysis of how often a runtime hypothesis is used and then leads to success within the same episode.
At the same time, in three of the four settings, StepLearn’s advantage is already present in Round 1 (Figure 4), and Figure 3 shows limited improvement across rounds in most settings. It is therefore unclear to me how much of the gain comes from step-level learning, and how much from knowledge accumulated on earlier tasks. Related to this, I am also not entirely sure what the authors intend to claim as the main evidence of effective test-time learning: is it sufficient that StepLearn outperforms the baselines in Round 1, or should it also show improvement across rounds on the same task?
To help separate the sources of the gains, could the authors provide some additional results? For example: 1. running each task’s 5 episodes independently (without sharing memory across tasks) for both StepLearn and the baselines; 2. disabling StepLearn’s persistent memory so that only step-level learning (within an episode) remains.
The reliability of the validation mechanism.
The “trust later” component relies on matching hypotheses to actions, but as far as I understand, candidates are matched only by scope and action type (§3.4), while the natural-language condition and policy are not used. If so, a hypothesis might be promoted or downgraded based on an action that does not follow its policy, e.g., clicking a different element on the same page type. In addition, the promotion thresholds (, ) are described as “a heuristic evidence gate” (Appendix A.3), and it is not entirely clear to me how these numbers were chosen. Since prequential validation is a key part of the contribution, it would be helpful to better understand how reliable it is in practice.
Questions for authors
-
Could the authors clarify how “scope” and “action type” are used for matching, and how accurate the promotion mechanism is in practice? For example, how often are promoted hypotheses validated by actions that are actually consistent with their policy?
-
Under the current experiment setup in the paper, a round consists of a stream of different tasks, and the stream is repeated for 5 rounds. What do the authors view as the main sign of effective test-time learning: learning across tasks within a round, or learning across rounds on the same task?
-
The central claim of the paper is to perform step-level learning at test-time rather than doing it only at episode-level, and the prequential validation seems designed to make step-level knowledge safe to reuse across episodes, rather than to improve cross-episode learning itself. Could the authors clarify whether StepLearn is also expected to be better at cross-episode learning? If so, could the authors explain in simple terms, why is StepLearn better at cross-episode learning compared to existing baselines (how does the step-level learning mechanisms translate to better cross-episode learning)?
-
StepLearn’s setup differs from existing TTL methods such as EvoTest and JitRL, which run all episodes of one task consecutively before moving to the next task. Could the authors clarify how these baselines were adapted to the round-based protocol to ensure a fair comparison?