- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes a famework for test-time adaptation through human-agent interaction ((TAHI) which uses session-level interaction signals into agent adaptation via two mechanisms: 1) Context-based: memory and skill induction and 2) Weight-based: fine-tuning (DPO, LoRA)
The motivation is that expert users have implicit domain knowledge — personal criteria and preferences — that cannot be specified upfront at design time. Their interaction with an agent during deployment implicitly reveals these preferences, and this signal can be used to adapt the agent at test time.
The authors conduct experiments on 30 users and 600 tasks, across 2 domains (abstract writing and data visualization) and show improvements over the base agent.
Critical strengths
The motivation of the paper is interesting and intriguing and identifies an underexplored gap in agent personalization. The premise that expert users hold implicit domain knowledge that cannot be elicited upfront, and that this knowledge surfaces only through interaction, is both well-motivated and practically important. The paper's framing of test-time adaptation as the right focus for learning individual user preferences — rather than relying on pre-deployment fine-tuning — is a consequential contribution to the emerging literature on personalized agents. The experiments show improvements over the baseline. The paper is well written, clearly motivated and easy to follow.
Critical weaknesses
W1: Lack of strong baselines. The paper only comapes against an unadapted baseline agent. It does provides no comparison against stronger baselines. A meaningful comparison which I would expect, for example, is a comparison against general-purpose harnesses with memory and skill-induction capabilities (e.g., Hermes), which represent the natural competitive baseline for context-based adaptation. Another baseline could be using a framework like Meta Harness (Lee et al 2026) or to refine the agent using a coding agent. Without this, it is unclear how much of the observed improvement is attributable to TAHI's design specifically.
W2: Higg experimental variance. The baseline scores differ substantially between the context and weight experiment conditions in Table 1 (81.7 vs. 82.8 and 77 vs. 69), although to my understanding these are the same baselines without adaptation on the same tasks. Although the authors show significance tests, this raises some concerns about experimental reliability and whether reported gains are meaningful.
W3: Evaluation generalization. The held-out evaluation tasks belong to the same two domains used during adaptation (abstract writing and data visualization), and differ from training data only in the year of the conference. This leaves open whether TAHI's adaptation signal really generalizes across task types. Demonstrating performance on qualitatively different tasks (e.g., coding, customer service, email writing) would be necessary to support the broader claims made in the paper.
Meta-Harness: End-to-End Optimization of Model Harnesses Yoonho Lee, Roshen Nair, Qizheng Zhang, Kangwook Lee, Omar Khattab, Chelsea Finn
Questions for authors
Q1: Table 1 baseline variance. The baseline scores differ substantially between the context-based and weight-based experiment conditions (81.7 vs. 82.8 and 77 vs. 69). Are these differences due to LLM non-determinism, different evaluation subsets, or another source?
Q2: Figure 4 rubric calibration. The traces in Figure 4 were edited by humans and are therefore presumed to contain agent errors — yet all rubrics score the majority of traces as successful, including the proposed method at 60%+. Without ground-truth annotations on these traces, it is unclear whether the rubric is failing to detect errors or whether agents genuinely succeeded on most steps despite human edits. Can the authors clarify how to interpret this figure, and provide any additional annotation that would help distinguish these two explanations?
Q3: Intra-user rubric consistency. As users iteratively refine their grading rubrics across sessions, do they consistently build on prior criteria, or do contradictions arise?
2 comments
Thank you for the response and clarifications. The strong similarity to Hermes could be made clearer in the paper - if the methods are so similar the Hermes harness should probably be cited. I understand the resource limitations regarding more tasks and domains. Overall I will maintain my scores.
We thank reviewer Hq4F for recognizing the interesting motivation, solid framing, and effective experiments of our paper.
W1. Lack of context-adaptation baselines.
We appreciate the reviewer for pointing out related context update methods. First, TAHI studies the general problem of using test-time human-agent interaction data to adapt an agent to a user's way of working. Methods like Hermes, Meta-Harness, and our memory+skill induction implementation are specific instantiations of context-based agent adaptation approaches.
Among these methods, we chose our context adaptation implementation given its top performance in preliminary experiments, which compares various context and weight adaptation methods in Appendix C. We found that switching the memory induction to Hermes performs similarly to our final context adaptation method in the offline setting (Hermes 84.3, TAHI: 84.9; overbaseline 80.8), because they share the same pipeline and merely differ in the LM induction prompts, yielding similar memory/skills content, which could potentially be extrapolated to other memory/skill induction methods. We will include these details of these preliminary context-adaptation comparisons in the revised paper.
Lastly, due to cost constraints, we conducted most method comparisons in preliminary experiments rather than comparing them all in the main online setting (Table 1), because a full online comparison would require recruiting 10 additional users and collecting 200 more tasks, which is very expensive given the expertise required in our setting.
W2 & Q1. Baseline experimental variances
Our baseline systems produce comparable solutions in context and weight adaptation settings. The variation in baseline scores primarily reflects differences in users’ rubrics: the same agent solution may receive 90/100 from one user but 60/100 from another with more demanding criteria. In fact, our experiments aim to show that adaptive agents outperform static baselines for each individual user, which accounts for this variance. Our statistical analysis (*s in Table 1) shows that our main finding holds.
W3. Evaluation generalization across domains.
We agree that evaluating TAHI across a broader range of tasks would be valuable. Yet formalizing the study on current tasks already requires substantial effort, as to the best of our knowledge, TAHI is the first work to adapt agents in real time with real human users’ diverse interactions.
Moreover, our evaluations require substantial resources, involved recruiting expert participants, collecting approximately 5,000 human-agent interaction turns, and maintaining 300 agent harness and weight checkpoints throughout the study. We believe these data and results provide evidence for TAHI’s promise. Extending the evaluation to a wider range of task types is an important direction for future work.
Q2. Rubric calibration and quality
We believe the score improvements reflect genuine capability gains rather than false positives in the rubrics. Figure 4 shows how each rubric scores low-quality solutions. Our rubric produces a wider score distribution, suggesting that it better distinguishes among failures. Figure 7 (Appendix E) shows scores for high-quality solutions evaluated with the same rubrics. Our rubric assigns these solutions higher scores, with scores concentrated near the top, suggesting fewer false-positive criteria. Together, these results indicate that our rubric can reward high-quality solutions while identifying issues in low-quality ones. By comparison, LLM-generated rubric scores are less distinguishable between the two sets of solutions.
Q3. Intra-user rubric consistency
They usually build on top of prior criteria, as they often express consistent preferences throughout task sessions.