acceptodds
Under review as a conference paper at ICLR 2027

Efficient Test-Time Adaptation through Human-AI Interaction

Abstract

AI agents are trained on population-scale data to encode broad capabilities spanning those of many practitioners. Yet the artifacts they produce rarely meet the personal bar professionals need to stake their reputation on. On realistic, open-ended tasks where success criteria are heterogeneous and insufficiently documented, individual expertise lives precisely in the elevation and departure from the average. In practice, iterative human-agent interaction surfaces criteria that users cannot fully specify up front, yet apply repeatedly across tasks. We argue this cross-session interaction data is a rich, underused signal for closing the gap to individual expertise. In this work, we propose test-time adaptation through human-agent interaction (TAHI), which integrates these signals into agent context and weights, and crystallizes each user's training and evaluation criteria via an evolving rubric module. We adapt agents to 30 individuals in two high-utility domains, writing and visual creation, on a total of 600 tasks. Our agents improve solo task success by 4.5-20.9% within only tens of tasks. Meanwhile, our evolving rubric module serves as a scalable annotation tool, creating evaluation rubrics that catch 16.0-22.3% more failures than those from LMs or humans alone. While agents are adapted towards individuals, we show these personalized agents also produce improvements in success of up to 8.8% that generalize across users.

Primary area:
foundation or frontier models, including LLMs
Keywords:
ai agent, test-time adaptation, human-ai interaction

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_Hq4FOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a famework for test-time adaptation through human-agent interaction ((TAHI) which uses session-level interaction signals into agent adaptation via two mechanisms: 1) Context-based: memory and skill induction and 2) Weight-based: fine-tuning (DPO, LoRA)

The motivation is that expert users have implicit domain knowledge — personal criteria and preferences — that cannot be specified upfront at design time. Their interaction with an agent during deployment implicitly reveals these preferences, and this signal can be used to adapt the agent at test time.

The authors conduct experiments on 30 users and 600 tasks, across 2 domains (abstract writing and data visualization) and show improvements over the base agent.

Critical strengths

The motivation of the paper is interesting and intriguing and identifies an underexplored gap in agent personalization. The premise that expert users hold implicit domain knowledge that cannot be elicited upfront, and that this knowledge surfaces only through interaction, is both well-motivated and practically important. The paper's framing of test-time adaptation as the right focus for learning individual user preferences — rather than relying on pre-deployment fine-tuning — is a consequential contribution to the emerging literature on personalized agents. The experiments show improvements over the baseline. The paper is well written, clearly motivated and easy to follow.

Critical weaknesses

W1: Lack of strong baselines. The paper only comapes against an unadapted baseline agent. It does provides no comparison against stronger baselines. A meaningful comparison which I would expect, for example, is a comparison against general-purpose harnesses with memory and skill-induction capabilities (e.g., Hermes), which represent the natural competitive baseline for context-based adaptation. Another baseline could be using a framework like Meta Harness (Lee et al 2026) or to refine the agent using a coding agent. Without this, it is unclear how much of the observed improvement is attributable to TAHI's design specifically.

W2: Higg experimental variance. The baseline scores differ substantially between the context and weight experiment conditions in Table 1 (81.7 vs. 82.8 and 77 vs. 69), although to my understanding these are the same baselines without adaptation on the same tasks. Although the authors show significance tests, this raises some concerns about experimental reliability and whether reported gains are meaningful.

W3: Evaluation generalization. The held-out evaluation tasks belong to the same two domains used during adaptation (abstract writing and data visualization), and differ from training data only in the year of the conference. This leaves open whether TAHI's adaptation signal really generalizes across task types. Demonstrating performance on qualitatively different tasks (e.g., coding, customer service, email writing) would be necessary to support the broader claims made in the paper.

Meta-Harness: End-to-End Optimization of Model Harnesses Yoonho Lee, Roshen Nair, Qizheng Zhang, Kangwook Lee, Omar Khattab, Chelsea Finn

Questions for authors

Q1: Table 1 baseline variance. The baseline scores differ substantially between the context-based and weight-based experiment conditions (81.7 vs. 82.8 and 77 vs. 69). Are these differences due to LLM non-determinism, different evaluation subsets, or another source?

Q2: Figure 4 rubric calibration. The traces in Figure 4 were edited by humans and are therefore presumed to contain agent errors — yet all rubrics score the majority of traces as successful, including the proposed method at 60%+. Without ground-truth annotations on these traces, it is unclear whether the rubric is failing to detect errors or whether agents genuinely succeeded on most steps despite human edits. Can the authors clarify how to interpret this figure, and provide any additional annotation that would help distinguish these two explanations?

Q3: Intra-user rubric consistency. As users iteratively refine their grading rubrics across sessions, do they consistently build on prior criteria, or do contradictions arise?

2 comments

We thank reviewer Hq4F for recognizing the interesting motivation, solid framing, and effective experiments of our paper.

W1. Lack of context-adaptation baselines.

We appreciate the reviewer for pointing out related context update methods. First, TAHI studies the general problem of using test-time human-agent interaction data to adapt an agent to a user's way of working. Methods like Hermes, Meta-Harness, and our memory+skill induction implementation are specific instantiations of context-based agent adaptation approaches.

Among these methods, we chose our context adaptation implementation given its top performance in preliminary experiments, which compares various context and weight adaptation methods in Appendix C. We found that switching the memory induction to Hermes performs similarly to our final context adaptation method in the offline setting (Hermes 84.3, TAHI: 84.9; overbaseline 80.8), because they share the same pipeline and merely differ in the LM induction prompts, yielding similar memory/skills content, which could potentially be extrapolated to other memory/skill induction methods. We will include these details of these preliminary context-adaptation comparisons in the revised paper.

Lastly, due to cost constraints, we conducted most method comparisons in preliminary experiments rather than comparing them all in the main online setting (Table 1), because a full online comparison would require recruiting 10 additional users and collecting 200 more tasks, which is very expensive given the expertise required in our setting.

W2 & Q1. Baseline experimental variances

Our baseline systems produce comparable solutions in context and weight adaptation settings. The variation in baseline scores primarily reflects differences in users’ rubrics: the same agent solution may receive 90/100 from one user but 60/100 from another with more demanding criteria. In fact, our experiments aim to show that adaptive agents outperform static baselines for each individual user, which accounts for this variance. Our statistical analysis (*s in Table 1) shows that our main finding holds.

W3. Evaluation generalization across domains.

We agree that evaluating TAHI across a broader range of tasks would be valuable. Yet formalizing the study on current tasks already requires substantial effort, as to the best of our knowledge, TAHI is the first work to adapt agents in real time with real human users’ diverse interactions.

Moreover, our evaluations require substantial resources, involved recruiting expert participants, collecting approximately 5,000 human-agent interaction turns, and maintaining 300 agent harness and weight checkpoints throughout the study. We believe these data and results provide evidence for TAHI’s promise. Extending the evaluation to a wider range of task types is an important direction for future work.

Q2. Rubric calibration and quality

We believe the score improvements reflect genuine capability gains rather than false positives in the rubrics. Figure 4 shows how each rubric scores low-quality solutions. Our rubric produces a wider score distribution, suggesting that it better distinguishes among failures. Figure 7 (Appendix E) shows scores for high-quality solutions evaluated with the same rubrics. Our rubric assigns these solutions higher scores, with scores concentrated near the top, suggesting fewer false-positive criteria. Together, these results indicate that our rubric can reward high-quality solutions while identifying issues in low-quality ones. By comparison, LLM-generated rubric scores are less distinguishable between the two sets of solutions.

Q3. Intra-user rubric consistency

They usually build on top of prior criteria, as they often express consistent preferences throughout task sessions.

Thank you for the response and clarifications. The strong similarity to Hermes could be made clearer in the paper - if the methods are so similar the Hermes harness should probably be cited. I understand the resource limitations regarding more tasks and domains. Overall I will maintain my scores.

Reviewer_m2moOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes TAHI, a framework for test-time adaptation through continual human-AI interaction. The method transforms user messages, direct edits to artifacts, and revisions to plans and evaluation rubrics into adaptation signals. For context adaptation, it constructs reusable memories and skills, while for weight adaptation, it converts interaction histories into preference data for parameter updates.

The experiments cover scientific paper abstract writing and HTML data visualization, including online task-by-task evaluation, held-out task evaluation after freezing the adaptation state, and ablation studies on interaction signals. The results show that both adaptation approaches improve the criterion pass rate under the rubrics used by the authors. Using the full interaction history also outperforms using user messages alone. The paper's main contribution is integrating multiple forms of natural interaction into a continual adaptation process and providing preliminary empirical evidence involving real users.

Critical strengths
  1. The paper proposes an interaction-driven adaptation framework with practical value. The paper transforms user messages, artifact edits, plan revisions, and rubric modifications into adaptation signals, supporting both in-context adaptation and weight adaptation. The offline in-context adaptation experiments in Table 2 show that using the full interaction history outperforms using messages alone, providing empirical support for leveraging richer forms of human feedback.

  2. The paper provides evidence of cross-task adaptation involving real users. The paper conducts experiments in two scenarios: abstract writing and data visualization. In the online evaluation, the agent first generates an output for the current task and then receives feedback on that task. The paper also includes evaluations on held-out tasks after freezing the adaptation state. These designs provide preliminary evidence that interaction experience can improve performance on subsequent tasks.

Critical weaknesses
  1. The incremental benefits over existing personalization methods have not been sufficiently validated. DITTO has investigated constructing preference data from a small number of user demonstrations and achieving personalization through parameter updates, which is closely related to the weight adaptation mechanism proposed in this paper. The current experiments lack direct comparisons with such methods under matched user data and base models. Although Table 2 supports the value of additional feedback information, it does not distinguish the contributions of increased information from those of the specific adaptation mechanism. Therefore, the additional value of the proposed approach over existing methods remains unclear.

  2. The validation of actual user benefits, rubric quality, and overall efficiency remains insufficient. The main results rely on the average criterion pass rate under evolving rubrics, without independent preference evaluations by target users on outputs from held-out tasks. Lower initial output scores do not directly demonstrate that the rubrics identify more genuine failures, as they may also introduce more false positives. Appendix E provides supplementary checks, but an independent human reference is still missing. Furthermore, although the paper reports inference token costs, the evaluation of human time and adaptation training costs is insufficient to support conclusions about overall efficiency.

Questions for authors
  1. How much additional benefit do the proposed mechanisms provide compared with existing personalization methods? Could the authors conduct controlled comparisons using the existing offline interaction records, while matching user data, base models, and tool environments? For in-context adaptation, baselines based on historical summaries or retrieval could be included. For weight adaptation, SFT and an adapted version of DITTO could be considered. Please clarify the adaptation details and report training and sampling budgets to distinguish the contributions of the proposed mechanisms, additional feedback, and increased computation.

  2. Do improvements in automatic evaluation correspond to user-perceived quality improvements and time savings? Following DITTO's blinded evaluations by target users and CollabLLM's evaluation of artifact quality, interaction experience, and task duration, could the authors ask the original target users to conduct anonymous pairwise comparisons between Baseline and Adapted outputs on held-out tasks? Please report win/tie/loss rates, per-user results, and uncertainty estimates, along with human time or intervention counts, to assess improvements in both quality and efficiency.

  3. Can evolving rubrics identify more genuine failures while controlling false positives? Following EvalGen's evaluation against human references, could the authors report failure recall (the proportion of human-confirmed failures correctly identified) and false failure rate (the proportion of human-accepted outputs incorrectly classified as failures) on a fixed set of outputs? Please freeze the evaluator and ensure that test labels are not used for rubric selection. Additionally, include a one-shot rubric generation baseline with access to the same interaction history to isolate the contribution of the rubric evolution mechanism itself.

  4. Could the authors provide results on an external benchmark? LongLaMP-2 includes a personalized scientific paper abstract generation task, which is relatively close to the writing scenario considered in this paper. Evaluating the adaptation components on its official splits would improve external comparability. However, this benchmark uses fixed user histories and cannot fully replace the interactive experiments in the paper. If the benchmark is not suitable, please clarify the differences and provide a reproducible evaluation protocol for the original tasks. This suggestion has lower priority than the controlled comparisons and independent evaluations discussed above.

2 comments

We thank reviewer m2mo for recognizing the practical value and realism of our work!

W1. Lack of direct comparison with existing methods

We would like to clarify that DITTO and TAHI are fundamentally different in three ways, so they cannot be directly compared in our setting.

  1. Different learning paradigms: DITTO and TAHI represent different paradigms thus require different training data: DITTO requires complete human references, TAHI only requires human feedback on agent-driven solutions, reducing the burden of data collection, especially under the increasing usage of agents.
  2. Agentic data format: DITTO targets single-turn LM generation and TAHI trains LM agents for multi-step generation from human-agent computer-use interactions.
  3. Test-time adaptation: after training, DITTO keeps the LM fixed throughout tests, whereas TAHI intends to adapt agents throughout each test session. The phrases of applying the approaches differ (DITTO at train, TAHI at test time) and thus the constraints in method designs differ (DITTO can flexibly batch training examples, while TAHI has to deal with effective learning from individual examples). Nonetheless, we share a lot of our motivation (tailor the system to individuals) and approaches (e.g., modify upon DPO). We thank the reviewer for pointing out this relevant work, and will add the discussions above to the revised version.

W2 & Q2. Do improvements in auto-eval match user perception

We did conduct further quality validation with real users. As described in lines 345-355, each of our participants manually specified their evaluation criteria, which we then used to evaluate a 30-task held-out set. As shown in Table 1 (right 2 columns), users substantially prefer adapted agents’ solutions over static ones. As we noted in footnote 6 (page 7), recruiting participants for additional validation was very costly, we therefore used their manual rubric annotation as a more scalable way to validate that our agent improvements match user perception. We will add these explanations to the revised paper version to convey our studies better.

W2 & Q3. Lack of validation on rubrics

We believe our agent score improvements reflect genuine capability gains rather than false positives in the rubrics. Figure 4 shows our human-interaction-derived rubrics produce lower scores, thus can more effectively identify low-quality solutions than LM-produced rubrics, suggesting our rubrics have fewer false positives, if any, than LM rubrics.

Moreover, Figure 7 (Appendix E) compares evaluations using our rubrics and LLM-produced rubrics, against final solutions that users deemed satisfactory. Our rubrics produce high scores, comparable to LLM rubric scores. Taken together with Figure 4, our rubrics can effectively distinguish high and low-quality solutions; in contrast to LM rubrics that cannot.

Q1. How much additional benefit does TAHI have over baseline methods

We compared different adaptation approaches in Appendix C, including different training algorithms (DPO, OPD, GRPO, REINFORCE) and different data construction approaches (e.g, DPO pairing, trajectory cleaning). DPO, our final approach, performs the best among our preliminary experiments.

First, TAHI targets a different learning paradigm from human-agent interaction data, which differs from prior single-turn LM personalization methods for single-turn LMs based on static references. We had to substantially modify existing methods and algorithms to make it applicable for our learning setting.

Second, we were constrained to conduct most methodological comparisons in preliminary experiments or in offline settings due to cost constraints of online agent adaptation (Table 1), since evaluating each additional approach online would require recruiting 10 additional users and collecting 200 more tasks, which is costly given the expertise required in our setting.

We understand that the current main content may lack discussions of baseline comparisons. We will add core discussions and results in Appendix C into the main content once more space is allowed in the revised version.

Q4. Results on external benchmark

We appreciate the reviewer’s suggestion of related resources. As we mentioned in response to W1, our setting requires datasets that feature agent trajectory solutions instead of single-turn responses, whereas the proposed LongLaMP-2 features single-turn responses. To the best of our knowledge, only PAHF [1] and its datasets would satisfy our three requirements (detailed in W1: interaction paradigm, agentic solutions, test-time learning), but they are not publicly available. We therefore collected our own high-quality dataset without using existing benchmarks. Our code for interface implementation, data collection, and evaluation will be open-sourced, so future work can reproduce and build upon our method.

w1/Q1: Authors' claim TAHI is efficient, but there is a lack of detail comparisons with simple baslines. I understand original DITTO is not comparable, but using offline dataset to compare with history summary, retrieval, and SFT is still neccessary. I want to know why are the specific mechanisms proposed by TAHI necessary, rather than using a simpler approach?

w2/Q2: Given the high cost of the experiment, we admit combination of human-written rubrics and history interaction summary could be used to indirectely evaluate users' preference. It is recommended that the author phrase the relevant conclusions more carefully.

W2/Q3:Existing experiments show that evolved rubrics can capture user requirements in the current interaction, but they do not yet provide sufficient evidence that they have learned generalizable user preferences.

Reviewer_J9rSOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This papers considers the use of multi-session interaction data to customize agent behaviour to individual users, introducing a method they refer to as test-time adaption through human-agent interaction (TAHI). Their method uses session data to construct per-user training and evaluation criteria. The work is motivated by the observation that generalist agents can improve their overall performance by adapting to the multi-session context of specific users. TAHI alongside corresponding experimental data is their main contribution. Their experiments test their system on 30 human users across 600 tasks in two domains (writing/visual creation). In their experiments, the authors observe improvements between 3.2 and 6.6% for context-adaptive, and 4.8 and 5.2 for weight-adaptive agents on held-out tasks.

Critical strengths
  1. High impact research area. The customisation of agents is an area of very active development inside industry, but limited academic analysis exists that can provide the public with a good understanding of the dynamics of agent personalisation. I think both making personalisation better but also improving public understanding of personalisation seem like a very valuable contribution. This work represents an important and timely contribution in this direction.
  2. Well executed. The paper is comprehensive in its experimental design, considering both weight adaption (i.e. model fine-tuning) and prompt-based adaptations. Typically papers propose either, but it's very useful to see a direct comparison. Experiments are small but with do mitigate their size with statistical analysis (though see questions regarding the validity of that analysis). Further, the writing is clear and easy to follow.
Critical weaknesses
  1. Small scale (only 30 people), though I consider it defendable given the high cost of data and mitigation with statistical analysis. I don't think it would be reasonable to ask for more data as it may not be feasible for many authors.
Questions for authors
  1. How did you decide on writing and visual creation as the domains to test on?
  2. Can you extend on your statistical tests (t-tests), clearly stating what the independent experiments are? My understanding is that some of these experiments may actually be highly correlated due to being from the same user, raising questions around the validity of the statistical tests.
  3. L21: Your abstract says you adapt to 30 people, but really its 20, given 10 are control?
  4. Missing related work: Could you elaborate how does your work relate to CIPHER (https://neurips.cc/virtual/2024/poster/96078) and PAHF (https://arxiv.org/abs/2602.16173)ß?

Further suggestions (no need to reply, do not affect score)

  1. L015: "Elevation and departure from the average", odd phrasing (also used similarly in L40), what does it mean to elevate from average??
  2. L339: The Tinker citation is odd (tin, 2025??)
  3. L371: "task , respectively" spurious space
2 comments

We thank reviewer J9rS for recognizing the importance of the direction and comprehensiveness in the execution of our project!

W1. Small scale of data

We believe our data has sufficient scale and well supports our claims in the paper. First, although it includes 30 users, each user completed 20 tasks with an average of 8 turns, yielding a total of 600 tasks and approximately 5k turns, which we believe has substantial scale. Second, based on the statistical significance analysis in Table 1, our data scale is sufficient to support all the claims that we made. Lastly, as the reviewer notes, collecting this data was costly and required substantial participant expertise. We invested huge effort (i.e., 200 hours of human annotation) in recruiting 30 experts across the domains studied.

Q1. How are writing and visual tasks selected?

We focus on open-ended tasks that (i) allow diverse solutions and evaluation criteria from individuals, (ii) are relevant to professional work, so our improved agents would have meaningful real-world impact. Beyond surface-level user preferences, we want tasks that require actual professional expertise to solve.

In this paper, we start with tasks that our researchers could oversee participant recruitment and assess task quality, and where current LMs possess varied levels of capabilities to enable meaningful comparisons between tasks (e.g., LMs are generally good at writing, but relatively weak in visual creation). Nevertheless, we believe our approach could extend to a broader range of tasks, and hope to extend this study further to other digital tasks.

Q2. Explain statistical tests on bias from the same user

We perform t-tests between (i) the scores averaged across 20 tasks for each user interacting with adaptive agents, and (ii) average task scores for each user interacting with static agents. Thus, each observation (i.e., each score) represents one user, and different observations do not share the same user. We believe this prevents biased observations from the same user.

Q3. Detailed participant count in experiments

Agents adapt to all 30 users but in different ways. For 20 users, we perform online agent adaptation, meaning that the agent’s context/weight changed during their 20 task sessions. For the other 10 users, we perform offline agent adaptation, meaning that the agents remain the same during 20 tasks, and adapt their context/weights afterwards (line 312-315). So 30 users were used to test the adaptation approaches, although some were evaluated offline.

Q4. Relationship with CIPHER and PAHF

We thank the reviewer for pointing to these related works. Both works share our goal of adapting agents to individual users, but TAHI differs in three ways:

  1. Broader interaction signals: TAHI captures diverse human interaction signals: plans, verification, messages (what RAHF primarily uses), and edits (what CIPHER primarily uses).
  2. Multiple adaptation methods: TAHI supports both context and weight adaptation, and performed extensive studies (Appendix C) to identify the best algorithm, whereas RAHF and CIPHER focus on context updates. This lets us show that weight adaptation can be as efficient and effective as context adaptation.
  3. Realistic expert interaction: TAHI puts special emphasis on realistic, expert human signals during interaction. We believe our data provides richer signals than fixed reference texts in CIPHER and static user preferences in PAHF.

Thank you for the suggestions on specific wording. We will refine them in the revised version.

Thank you for your response, I have some further questions.

W1. Small scale of data

As I said, I think it's not reasonable to ask for more data but I am surprised to see you claim that such a small experiment size is completely sufficient. I think this limitation should at the very least be explicitly acknowledged. Or would you say a larger experiment would not have provided more confidence?

Also the 5000 turns number is irrelevant to the statistical significance (since that is not the unit of any test), so I consider it not relevant to raise this number here.

Q2. Explain statistical tests on bias from the same user

Your answer confused me more than it helped, as it seems to contradict the paper's description.

Some followup questions:

  1. In the paper you state that you conducted paired t-tests on 100 datapoints (L371-372), in your response you say tests between per-user means (of the same users, or of two different user groups?). Which one is true?
  2. For each cell in Tab 1, can you please provide sample size n, degrees of freedom, the t statistic and the exact p-value, and the mean (over tasks) baseline and adaptive score of each of the 5 users ?
  3. In Table 1 caption you provide the |t| > 2.0 as signficiance threshold, this value corresponds to p<0.05 only for ~60 degrees of freedom or more, with 5 users per cell the value would be different. How did you choose 2.0 and how does it related to the stars shown in the table.

Further point: I noticed that the linked repo in the paper doesn't work, you put that status in stead of repo. Managed to find despite this.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.