acceptodds
Under review as a conference paper at ICLR 2027

Flexible Agent Alignment with Goal Inference from Open-Ended Dialog

Abstract

We introduce Open-Universe Assistance Games (OU-AGs), a formal framework extending assistance games to LLM-based agents. Effective assistance requires reasoning over human preferences that are unbounded, underspecified, and evolving. Existing assistance game formulations assume fixed, predefined preferences, an assumption that breaks down in open-ended dialogue where goals are revised incrementally and expressed in natural language. Grounded in cognitive science accounts of preference construction, we represent human preferences as a dynamically updated distribution over discrete natural-language goals. To operationalize OU-AGs, we introduce GOOD (GOals from Open-ended Dialogue), a data-efficient online method that extracts and ranks candidate goals during interaction, using LLM-simulated users to perform probabilistic inference over goal hypotheses, allowing for interpretable, uncertainty-aware preference representations without large offline datasets. Across text domains in grocery shopping, household robotics (AI2-THOR), and coding, GOOD matches or exceeds baselines without explicit goal tracking in alignment with user intent, with the largest gains in robotics and coding, while producing interpretable goal updates justified by the dialogue. On existing multi-turn preference benchmarks, where the agent's choices have limited influence on how the interaction unfolds, explicit goal tracking helps less, suggesting that its value lies in letting inferred goals shape the agent's actions.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
evolving human preferences, LLMs, assistance games, human-ai collaboration, dialogue systems

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_B4dgOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how agents can infer and adapt to user goals as requirements are progressively clarified or revised during open-ended, multi-turn dialogue. It introduces Open-Universe Assistance Games (OU-AGs), extending assistance games from fixed preference types to dynamically evolving goal sets. To operationalize this framework, the authors propose GOOD (GOals from Open-ended Dialogue), which uses LLMs to propose, prune, and rank natural-language goal hypotheses. Its probabilistic variant aggregates pairwise comparisons into Beta-based confidence estimates, which guide action selection and further clarification without additional model training.

The authors evaluate GOOD through limited-scale interactive experiments in grocery shopping, AI2-THOR household robotics, code generation, and simplified Stateful SWE tasks, alongside subsets of Lost in Conversation, PrefEval, and GoalBench. Interactive evaluations primarily use simulated users and LLM judges, supplemented by a small human-rating study. GOOD matches or exceeds full-context and summary baselines in some interactive settings and model configurations, with larger gains in robotics and coding. At the same time, improvements on the external benchmarks are limited and inconsistent. These results suggest that explicit goal tracking may be particularly useful when inferred goals can influence subsequent questions and actions.

Critical strengths

1. A useful formulation of evolving user intent. The paper addresses an important challenge: user requirements are often incomplete and become clearer or change during interaction. OU-AGs and GOOD provide an explicit, inspectable representation of candidate goals, making it possible to examine how inferred intent influences clarification and action selection.
2. Promising evidence in interactive settings. GOOD improves performance in several interactive task configurations, particularly robotics and coding. The small human-rating study also supports improvements over the full-context baseline in selected grocery and robotics experiments. The more mixed results on external benchmarks provide useful evidence about the method’s potential scope.

Critical weaknesses

1.The incremental contribution of probabilistic goal tracking is insufficiently isolated. Full-context and summary baselines do not establish whether the gains require probabilistic inference or primarily come from organizing user requirements and allocating additional computation. GOOD (prompt) provides a relevant partial control, but it also differs in pruning and planner inputs. The paper does not report comparisons with independent structured-memory methods or integrations into representative frameworks such as ReAct or Plan-and-Execute, despite claiming compatibility with existing action agents. This limits the evidence for GOOD’s distinctive contribution.

2.The evidence for reliable uncertainty-aware assistance remains limited. The Beta-based scores aggregate LLM comparison outcomes, but their calibration as confidence in user goals is not established. Candidate coverage is another unresolved limitation: Appendix A.1 shows strong ranking performance when the correct answer is guaranteed to be present, but no clear advantage when candidates must be generated. This analysis uses only 30 questions and does not establish how coverage failures affect downstream interaction or recovery. The small interactive task sets—particularly five simplified Stateful SWE problems—and reliance on simulated users and LLM judges further limit support for broader claims.

Questions for authors

1. Confidence under correlated evidence. The implementation accumulates pairwise judgments based on overlapping dialogue histories without correcting for their dependence. What justifies interpreting the resulting reduction in Beta variance as increased knowledge of user intent? Is there evidence that confidence remains reliable when additional comparisons provide little new user information?
2. Adaptation to preference revisions. Retained candidates preserve their historical win/loss counts, without an explicit forgetting or change-detection mechanism in the main update path. How quickly does GOOD revise a previously confident goal after the user contradicts it? Can the authors provide evidence distinguishing adaptation to genuine preference changes from ordinary refinement?
3. Recovery from inadequate candidates. The implementation proposes new candidates each round and retains an undefined Baseline hypothesis. How reliably do these mechanisms distinguish an inadequate candidate pool from a sufficiently supported goal? Is there evidence showing that GOOD detects and recovers from coverage failures before taking incorrect actions?

3 comments

Thank you for the careful review. We are glad you found that OU-AGs and GOOD give an explicit, inspectable representation of evolving user intent, and that the interactive results, especially in robotics and coding, and the human-rating study are promising.

Calibration and correlated evidence: The reviewer is correct that the Beta mean is a relative win rate among candidates, not an exact calibrated probability, we treated it as a ranking and gating signal that is more principled and allows for interpretability rather than just one-shotting an LLM model end to end. Goals that pass the "certain" gate are correct 84% of the time, though ranking helps much more in coding than in grocery, where most candidates are already acceptable. Following your suggestion we tested a variant that decays old evidence (wins and losses multiplied by 0.9 each round), compares new goals against the current leaders, and caps the pool size (and removes a goal when its Beta mean is below 0.25 and it has at least 4 decayed wins+losses):

GOOD (prob), same held-out personasOriginalWith decay
Precision of the "certain" gate84% [72, 94]94% [89, 98]
Rounds acting on a wrong "certain" goal21% [9, 35]4% [1, 8]
Top-ranked goal correct82% [68, 94]94% [89, 97]
Rounds with no goal certain (agent keeps asking)11% [9, 14]38% [26, 47]

The decay makes GOOD more cautious: it acts on a wrong goal far less often, at the cost of asking in more rounds. For the camera ready of the paper, we will run all of the experiments with the decay.

**intent switching experiments: ** Our main runs cover cases where users reveal preferences gradually, so based on your suggestion, we run an additional intent-switch experiment. Partway through each grocery or coding episode, the simulated user changes a stated preference (a dietary constraint after the matching items are in the cart, or a style requirement after the code is written). A judge scores whether and how quickly the agent acts on the new preference, what fraction of the items the new preference requires end up in the final cart or code, and how well the final outcome matches the new preference overall (5 personas × 3 trials per domain, mean ± SEM):

GOOD (prob) acted on the reversal in every run, while Full Context missed it in 60% of coding and 32% of grocery runs. Summary Context is the fastest to respond to the reversal in coding (0.7 rounds), but its final code satisfies the revised requirement much less well (final alignment 6.2 vs. 9.5), and GOOD (prompt), which tracks the same goals without the probabilistic update, trails GOOD (prob) in both domains (7.7 vs. 9.5 alignment in coding; 64% vs. 91% coverage in grocery).

DomainMethodAdapted (%) ↑Rounds to adapt ↓New-requirement coverage (%) ↑Final alignment (0–10) ↑
CodingGOOD (prob)100 ± 05.1 ± 1.584 ± 59.5 ± 0.4
GOOD (prompt)93 ± 78.4 ± 1.684 ± 47.7 ± 0.8
Summary Context100 ± 00.7 ± 0.478 ± 66.2 ± 0.9
Full Context40 ± 138.3 ± 3.083 ± 86.1 ± 0.9
GroceryGOOD (prob)100 ± 01.1 ± 0.291 ± 59.2 ± 0.4
GOOD (prompt)100 ± 00.8 ± 0.164 ± 118.2 ± 0.7
Summary Context100 ± 00.9 ± 0.282 ± 69.4 ± 0.3
Full Context68 ± 123.3 ± 0.981 ± 79.3 ± 0.3

Isolating the probabilistic goal tracking; planning module: GOOD (prompt) is our closest control: it uses the same goal proposals and planner, and replaces the pairwise Beta inference with a prompted choice of the most likely goal. In the new intent-switch experiment (see Q2), GOOD (prob) beats GOOD (prompt) on final alignment in coding (9.5 vs. 7.7) and on coverage of newly required items in grocery (91% vs. 64%). On integration with existing agents: GOOD's action module is already an act–observe–replan loop over domain-specific actions, which takes the current goal beliefs as input each round, so using ReAct or Plan-and-Execute changes only the planner that consumes those beliefs.

**Recovering from inadequate candidates: ** A goal list with no correct candidate is rare and short-lived (58 of 590 rounds, mostly the first round of an episode), because new candidates are proposed every round and a correct one enters after about one round on average. In those rounds Baseline ranks first 60% of the time, against 5% otherwise (97% vs. 9% with decay), and GOOD asked the user a question before its first incorrect action in every run. Recovery therefore comes mainly from proposing new candidates.

Dear reviewer,

Thank you for your time and thoughtful feedback on our submission. We have posted our rebuttal response and would be grateful if you could take a look at our response when you have a chance. Thank you again for your help in improving our work.

Thank you for the response. However, the lack of comparisons with a sufficient number of strong baselines remains a major issue.

Reviewer_Sn4hOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the problem of goal inference in open-ended, multi-turn dialogues. Existing LLM-based agents often struggle to reliably track shifts in user intent throughout a conversation. To address this, the paper introduces the "Open-Universe Assistance Games" (OU-AGs) framework, which models human preferences as a probability distribution over natural language goals that are dynamically updated. To optimize OU-AGs, the authors propose the "GOals from Open-ended Dialogue" (GOOD) method. GOOD efficiently extracts goals during an interaction by proposing, pruning, and ranking candidate goals, which are then used to guide the agent’s actions. In both robotics and coding domains, GOOD outperforms context-based baselines and enables interpretable goal updates driven by the dialogue.

Critical strengths
  1. The experimental section covers various practical domains such as shopping, robotics, and coding, demonstrating that the proposed method possesses a degree of cross-domain applicability.
  2. Inferring user intent is a problem of significant practical value. The paper identifies the core challenges as the ambiguity of user needs, the gradual revelation of preferences, and the potential for goals to evolve during the interaction process. By leveraging Assistance Games to model this goal uncertainty within the interaction framework of LLM agents, the paper offers a fresh research perspective.
Critical weaknesses
  1. The paper argues that the core challenge in goal inference lies in the fact that human goals evolve continuously throughout an interaction. However, experiments in interactive domains rely on predefined user personas rather than dynamic goals, and they do not evaluate the cases of goal revision (Line 155), which creates a mismatch between the experimental design and the core research motivation.
  2. The reliability of the experimental results is open to question. The experiments in interactive domains are limited in scale, utilize LLMs to simulate users, and employ LLM-as-a-judge approaches for evaluation, lacking objective success rate metrics, though the authors conduct a small-scale human evaluation to validate the judge.
Questions for authors

Q1: Could you include experiments on "human goal revision” scenarios where a user’s goals shift during a conversation (e.g., changes in purchasing requirements or programming specifications)?

This is a crucial addition, as it directly addresses the paper’s core motivation: that human goals evolve continuously throughout an interaction.

Q2: Could you enhance the reliability and authenticity of the experiments in interactive domains? For instance, by involving real human users and incorporating objective evaluation metrics?

In interactive settings, the current benchmarks contain only 5–10 tasks each, which is likely insufficient to cover the full spectrum of potential scenarios. For evaluating the ability to understand user intent, you could employ objective metrics, similar to those used in UserBench (https://arxiv.org/abs/2507.22034), to verify whether the final selection aligns with the pre-defined correct option and to quantify user preference. Furthermore, I suggest using real human participants rather than relying on LLM simulations, as this would significantly improve the reliability of your experimental results.

3 comments
AuthorsOpenReview ↗

Intent switch experiments and reliability

Thank you for the review. We are glad you found the problem practically important and the assistance-game perspective to be fresh.

Intent switching. Our main runs cover cases where users reveal preferences gradually, so based on your suggestion, we run an additional intent-switch experiment. Partway through each grocery or coding episode, the simulated user changes a stated preference (a dietary constraint after the matching items are in the cart, or a style requirement after the code is written). A judge scores whether and how quickly the agent acts on the new preference, what fraction of the items the new preference requires end up in the final cart or code, and how well the final outcome matches the new preference overall (5 personas × 3 trials per domain, mean ± SEM):

GOOD (prob) acted on the reversal in every run, while Full Context missed it in 60% of coding and 32% of grocery runs. Summary Context is the fastest to respond to the reversal in coding (0.7 rounds), but its final code satisfies the revised requirement much less well (final alignment 6.2 vs. 9.5), and GOOD (prompt), which tracks the same goals without the probabilistic update, trails GOOD (prob) in both domains (7.7 vs. 9.5 alignment in coding; 64% vs. 91% coverage in grocery).

DomainMethodAdapted (%) ↑Rounds to adapt ↓New-requirement coverage (%) ↑Final alignment (0–10) ↑
CodingGOOD (prob)100 ± 05.1 ± 1.584 ± 59.5 ± 0.4
GOOD (prompt)93 ± 78.4 ± 1.684 ± 47.7 ± 0.8
Summary Context100 ± 00.7 ± 0.478 ± 66.2 ± 0.9
Full Context40 ± 138.3 ± 3.083 ± 86.1 ± 0.9
GroceryGOOD (prob)100 ± 01.1 ± 0.291 ± 59.2 ± 0.4
GOOD (prompt)100 ± 00.8 ± 0.164 ± 118.2 ± 0.7
Summary Context100 ± 00.9 ± 0.282 ± 69.4 ± 0.3
Full Context68 ± 123.3 ± 0.981 ± 79.3 ± 0.3

Reliability, objective metrics and real users: Unlike UserBench, our tasks are open-ended and personas are vague (to introduce ambiguity over rounds), with many valid outcomes per persona, so there is no single correct outcome; we use an LLM judge instead, whose method rankings match those of 120 human annotators on outcome scores (Appendix A.10). We do compare goals to human annotated goals (GoalBench experiments). Simulated users keep each persona's preferences fixed across methods, which makes the comparison controlled. We list simulated users, LLM judges and the small human study as limitations; in the revision we will state them more prominently, together with the number of personas per domain. We do agree with the reviewer that larger human studies would be interesting to look at and think about careful design for to maintain naturalistic interactions in a controlled setting, we have mentioned larger human studies in future work.

Thank you for the response and the revision.

For Q1, The intent-switch experiment directly addresses the gap between the motivation and the original experiments, and the result that Full Context misses the switch in 60% of coding and 32% of grocery runs while GOOD (prob) always reacts is a useful finding.

For Q2, I accept the argument that open-ended tasks with vague personas have no single correct outcome, and the human study (120 annotators, consistent method ranking with the judge) gives reasonable support for the judge in grocery and robotics. I remain concerned that the coding results and the Table 5 metrics rely on the LLM judge alone, and Table 3 shows that absolute scores change substantially with the judge. The revised limitations and future work state this openly, which I appreciate. An objective check for coding (e.g., test pass rate on the persona requirements or Stateful SWE issues) would strengthen the paper further.

Generally, my two main concerns have been addressed, and I am raising my score.

Thank you for reading our rebuttals and thank you for your time.

Reviewer_F5NKOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper has two main contribution: The proposal of the Open-Universe Assistance (OU-AG) game framework, an combination of open-universe POMDPs and assistance games in the LLM context. The operationalize OU-AG through their algorithm GOOD, which employs LLMs as proposal function evolving (by adding, removing, and ranking) a discrete set of textual descriptions of inferred goals, which the agent could in turn use to inform their next action (if the environment allows this). They evaluate GOOD on two multi-round domains: The fixed-action domains where the environments outcome is pre-determined, regardless of the agents action, and the interactive domain, where the action can influence influence the revealed information. The compare performance to the benchmarks' original baseline and an LLM with a summary context. In the Constrained-Interaction Benchmarks GOOD's overall matches or performs slightly weaker than the baselines. The authors hypothesize that this is due to GOOD not being able to use it's inferred goals effectively due to the fixed outcome setup. In the interactive domain, at least one of the multiple tested GOOD variants seems to be on the Pareto front, although the main results are reported with respect to avg. cost per round, not total cost. The paper also features study on the interpretability of the tested approaches, as judged by an LLM and partially verified by human annotators.

Critical strengths
  • The authors address an important issue in human-AI interaction: Users preferences evolve during the interaction and effective methods for preferences-structure / intent-prediction need to keep track of that.
  • The proposed framework is well motivated and minimal. Section 3 does a good job of highlighting the constituents contribution and clearly outlining the changes made to form the new framework.
  • The use of LLMs as proposals within their framework and the link to particle filters is an interesting and promising one for human-AI interaction
  • The presentation is generally good and the plots are clear.
Critical weaknesses
  • I believe that some of the results regarding GOOD are overstated or presented in a misleading way.
    • Sec 6.1: "at least one configuration of GOOD matches or exceeds baseline performance on task performance at a modest increase in average cost per round": The authors present that conclusion based on figure 5, where the plot avg. round cost, a per-round statistic, vs. task performance, an outcome-level metric. While the cost increase for avg. cost per round seems to be more in the 2x to 3x range (eyeballed), which depending on the context might not be only "moderate", the plot hides the actual total cost of GOOD. This can be found in figure 10 (appendix), and there the cost increase across whole interaction episodes is exacerbated. This seems to be because GOOD generally takes many more rounds that the baselines. So the takeaway, to me, is that GOOD allows for goal tracking at a significantly higher cost - and not only token cost, but human cost too (which is arguably more costly).
    • Sec 6.1: "at least one configuration": The performance/cost of the configurations vary strongly across tasks, and to my eye, no clear pattern among the GOOD variants emerges. It would prove difficult for as a practitioner to chose a GOOD setup.
  • The human study is not described with enough detail. How many annotators were employed. Do we have any demographics on the annotators, including familiarity with the domain? See my questions below.
  • Any statement with respect to the GoalBench can not be verified, as the dataset seems to be under review and the GoalBench code has not been shared.
  • Overall, the authors present a promising framework but in an effort to sell the accompanying method GOOD as performant, the seem to force a comparison that forces them to choose misaligned metrics. What is the most important (in my opinion) contribution of this work is making goal tracking inspectable and thus potentially interpretable, but the evidence for this fact is weak, as it is based on a weaker LLM judge than the method uses and an intransparent human validation study.
Questions for authors
  • How do the authors justify choose the performance vs. avg. cost. per round as a headline? Can the authors reproduce plot 5 with total cost as the x-axis as well?
  • Can the authors share the specifics about the human annotation study, and do the authors consider running the annotation procedure that matches the GOOD models?
  • Could the authors also report rank-correlation for the annotated items?
  • What are the implications of using GOOD on human cost (attention, time-spent) in comparison to the baseline methods?
5 comments
AuthorsOpenReview ↗

Response to review

Thank you for the detailed and constructive review. We are glad you found the OU-AG and GOOD framework well motivated, and that you think that using LLMs to propose goals and the connection to particle filters. 

** Total cost and cost to the user:** We agree total cost matters and will add total-cost versions of Fig. 5 with rounds per episode (we reported cost per round because methods run for different numbers of rounds). GOOD costs more per episode, in money and in the user's time, but some of that cost is unavoidable: preferences are underspecified at the start and only come out as the conversation goes on (the baselines on the robot domain only have 4-6 rounds but score lower), and the fixed-interaction benchmarks show that goal tracking helps little when it cannot shape what the agent asks next (Tables 1, 2). GOOD also asks only while its belief is below a tunable confidence threshold (Fig. 14): in Lost in Conversation it asks 0.88 clarifying questions per conversation, against 1.40 for Summary.

How to choose a model trio setup: We agree this is important, thanks for calling this to our attention. We will add a short description to Sec. 6.1, our results show that a cheap/lighter model for goal ranking/proposition (the most frequently called module, ie: 5.4-nano with no reasoning) with a stronger model for the planner (a model with medium or high reasoning, ie: 5.4-mini with high reasoning) 

Details of the human study: Thank you for bringing this to our attention, we will expand Appendix A.10 with these details and on the rank and correlation: the study used 120 annotators recruited through Prolific (25 per trial for grocery and 15 per trial for robotics, over 3 trials), paid at least minimum wage under an IRB exemption. Annotators were adults (18–80) living in the US or UK, native English speakers with no reported literacy difficulties and at least a secondary education. We will fix our descriptions during revisions: that the two GOOD variants are within 3.5 points of each other, and humans and the judge sometimes order them differently, but that the humans and the judge agree on the main comparison: GOOD variant averages scores above Full Context for both raters, in both domains and on both outcome and conversation scores (the mean of the two GOOD variants):

Score (%)Human: GOODHuman: Full ContextLLM: GOODLLM: Full Context
Grocery cart75.865.182.776.2
Grocery conversation74.168.579.776.9
Robot actions62.743.862.029.1
Robot conversation87.966.588.084.6
The human preference for GOOD is significant in every row (paired Wilcoxon; grocery cart p = 0.005 and conversation p = 0.04; robot actions and conversation p < 10^{-7}. At the item level, the judge's cart scores correlate with the human means at Spearman ρ = 0.58 (n = 84, p < 0.001). 

Judge strength and the evidence for interpretability: Thank you for these concerns. We will revise the paper to clarify that the judge's role here is to compare methods: every method's runs are scored by the same judge with the same rubric, so what matters is whether the judge orders the methods consistently, and a weaker judge can do that as long as its rankings agree with humans generally (Appendix A.10). Inspired by this, we also rescore the interpretability metrics for all grocery, robotics and coding runs with a stronger judge, gpt-6-luna (low reasoning). The ordering of methods holds overall in the domain, though the stronger model is stricter. 

GoalBench: GoalBench (https://openreview.net/forum?id=nu9viwOlzg; code in its supplementary material) contains goals written by third-person human annotators across real human-AI conversations, and scores a model's inferred goals by weighted recall and precision against these human goals using an autograder calibrated on human similarity judgments. We use it to check that on 15 of its conversations, GOOD's inferred goals reach about the same precision and recall as goals elicited directly from the same model (Fig. 3).

Reviewer_F5NKOpenReview ↗

Response to the authors' rebuttal

Thank you for your detailed response, the added detail for the human study in particular.

I don't quite agree with the reasoning of the authors that per-round costs aids comparability, in my opinion it hides the core trade-off that is at the heart of the approach. The methods would have been comparable by reporting total cost. I do appreciate the authors willingness to promote this comparison to the main text.

To be concrete about what would change my assessment:

  • The revised paper moves total-cost comparison plot (along the per-round results) into the main text discusses the trade-off transparently in section 6.1 (on grocery: GOOD costs several times the full context baseline and does not beat it)
  • I would like to have the gpt-6-luna scoring reported in numbers, not just asserted

If you upload a draft revision during the discussion period I am happy to reassess my score on this basis.

AuthorsOpenReview ↗

PDF revisions + follow up on judge strength for interpretability

Thank you for your response to our rebuttal response. We have uploaded a draft revision where we moved the total-cost comparison task performance graphs into the main text and discuss the cost trade-off; model selection details; and we also revised with updates to judge strength/interpretability, which are described below.

Follow-up on judge strength for interpretability. Since our first response, we (1) fixed the trace given to the interpretability judge, whose "action taken" line showed each method's planner input (e.g., GOOD's goals, Summary Context's summary) rather than its executed actions, and raised the judge's output limit from 500 to 4,000 tokens (stronger judges write longer explanations, which the 500-token limit cut off before the structured score could be parsed, disproportionately for GOOD's longer traces), and (2) rescored all runs in all four domains with two judges from other model families, claude-haiku-5.5 and qwen3.8-flash, alongside gpt-4.1-mini and gpt-6-luna. Corrected trace, all domains and configurations (%, mean ± SEM across cells, as in Fig. 4):

gpt-4.1-minigpt-6-lunaclaude-haiku-5.5qwen3.8-flash
Interpretability (Goal–action coherence)
GOOD (prob)97.1 ± 0.855.1 ± 3.646.0 ± 1.355.5 ± 2.7
GOOD (prompt)95.8 ± 0.950.9 ± 2.638.5 ± 1.045.0 ± 1.5
Summary Context86.6 ± 8.252.8 ± 2.736.9 ± 2.034.9 ± 2.6
Full Context81.9 ± 11.447.4 ± 4.630.0 ± 2.130.5 ± 2.8

On interpretability, GOOD (prob) scores highest under all four judges, and the two GOOD variants are the top two under three of the judges; under gpt-6-luna the margin is within one SEM and Summary Context ranks second. The judges largely agree on how the methods rank (Kendall's W = 0.82; main configuration, mean over domains). Fig. 4 in the revised draft is now revised with Table 3.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.