acceptodds
Under review as a conference paper at ICLR 2027

ReferClaw: Disambiguating User Intent for Egocentric Assistance

Abstract

Users of egocentric AI assistants often ask short, ambiguous questions whose intended referents cannot be uniquely resolved from video and dialogue history. For example, "Where did I leave it?" may refer to several previously handled objects. Existing VideoQA systems typically commit to a single interpretation, risking plausible but intention-misaligned answers. We introduce ReferClaw, an agentic framework that formulates egocentric streaming QA as an interactive process: the assistant asks clarifying questions to resolve user intent before answering. Three cooperating agents identify candidate interpretations, request clarification, and generate answers grounded in the resolved intent and visual evidence. A bounded, cost-aware policy balances expected information gain against user effort while limiting clarification turns and question complexity. ReferClaw learns from interaction through intent-specific skill updates and process-reward-guided policy optimization with agent-level credit assignment. Evaluations on augmented egocentric QA benchmarks demonstrate improved answer accuracy and joint answer-and-grounding performance over existing baselines, alongside zero-shot transfer across datasets. These findings support interactive clarification as a practical approach to resolving online user intent in egocentric streaming assistance.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Egocentric AI Assistance, User Intent Disambiguation, Interactive Clarification, Self-Evolving Agent

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_Fc1xOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

ReferClaw helps an egocentric video assistant clarify ambiguous requests before answering. Three agents identify possible interpretations, choose a short clarification question, and answer from the selected evidence. The system also learns reusable skills and updates agent-specific adapters from feedback. Experiments cover augmented video QA datasets, image-based transfer, and human respondents.

Critical strengths
  1. The method addresses a real failure of video QA which is an answer can match the video but refer to the wrong encounter. Sharing grounded candidates between clarification and answering is a sensible way to address this.

  2. The controls are convincing. Frozen Base reaches 66.2% accuracy versus 60.5% with asking disabled at the same entering state. The paired-intent test also shows that answers follow the user's clarification rather than a fixed interpretation.

  3. The paper considers user effort as well as accuracy. It includes short-question budgets, unnecessary-asking measurements, transfer tests, and a human comparison in which ReferClaw's questions are answered faster and cause less confusion.

Critical weaknesses
  1. The evaluation relies on fairly ideal conditions. Learning receives the correct intent and annotated evidence after every prediction, while participants in the human study are told which intent to hold. These results are useful, but it remains unclear how well the system works when users give brief corrections, forget details, or express their own intentions naturally.

  2. The separation between validation and evaluation needs clarification. Appendix B.7 uses 1000 examples to decide whether to accept model updates, and Table 13 reports results on a 1000-example held-out set. Are these different sets? If the same examples are used for both purposes, the reported learning gains may be overly optimistic.

Questions for authors
  1. Are the reporting probe and adapter-promotion probe separate, and are both video-disjoint from the online stream?

  2. What learning gains remain with sparse correctness feedback or user corrections, without gold intervals and boxes?

  3. How were participants assigned across methods, and how consistent are the human-study gains across participants?

4 comments
AuthorsOpenReview ↗

Response to Reviewer Fc1x (1/3)

We sincerely thank the reviewer for the valuable time spent on our paper and the constructive comments. We appreciate the reviewer's acknowledgement of the significance of our problem setting and controlled analysis. We address the major concerns below and also reflect them in the revision (blue). Table, section, and appendix numbers refer to the PDF.

W1. Fairly ideal conditions

We understand the concern. In short: when users give brief corrections, forget details, or reply in their own words (simulated), learning still gains 5.3–6.1 points, and when participants express their own intentions (small pilot), the margin over Clarify-or-Answer stays similar.

a. Learning without the correct intent and annotated evidence

In the default setting, learning receives the correct intent and annotated evidence after every prediction, which follows MetaClaw's online protocol (Sec. 4.1) and common practice in learning from interaction. We remove them one at a time under the same stream and update rules (new in App. D.3). The table below gives the accuracy of ReferClaw (Full) in each setting (Tab. 8):

SettingRevealed after each predictionReferClaw (Full)ReferClaw (Full), probeGain over Base
Base (frozen)–66.265.80.0
S0 (default)intent + interval + box + verdict74.375.2+8.1
S1intent + interval + verdict74.074.9+7.8
S2intent + verdict73.474.1+7.2
S3verdict71.572.0+5.3
S4clarification reply only68.968.8+2.7

Gain: accuracy gain over frozen Base (66.2%), in points. Probe: accuracy on the held-out reporting probe (W2/Q1).

  • Without annotated evidence (S2), ReferClaw (Full) reaches 73.4%, because given the revealed intent, the system grounds the evidence itself.
  • Without the correct intent as well (S3), binary verdicts alone add 5.3 points (95% CI [+4.2, +6.4]).
  • With only the clarification reply (S4), learning still adds 2.7 points, since on asked inputs (18.6%) the candidate selected by the user's reply serves as a self-label, matching the correct intent on 81% of them.

b. Users who give brief corrections, forget details, or reply in their own words

We thank the reviewer for suggesting these settings. We simulate these users on 50% of user utterances:

  • Brief: at most three words, e.g., "not that one" or "earlier".
  • Forgetful: the distinguishing detail is removed from the simulated user's target description before it replies.
  • Free-form: the user answers in their own words without following the options, generated by Llama-3.3-70B to avoid the style of the main simulator.

The first table below applies these noise types to the user's corrections during learning (verdict + one-sentence correction, Tab. 11):

CorrectionsReferClaw (Full)ReferClaw (Full), probeGain over Base
Clean72.673.2+6.4
50% brief72.172.6+5.9
50% forgetful71.572.0+5.3
50% free-form72.372.8+6.1

Probe and Gain: as in §a.

The second table applies them to the user's replies to the assistant's clarification questions, with frozen Base and no learning (Tab. 21):

RepliesAnswer accuracyClarification successTurns per candidate-bearing inputCost per asked inputAbstention
Clean66.277.50.470.451.6%
50% brief65.675.40.490.461.9%
50% forgetful64.371.20.550.513.2%
50% free-form65.174.00.500.472.1%

Cost: normalized respondent cost of the questions on an asked input (App. B.2).

  • Forgetful users are the hardest case for learning (71.5%) and clarification (64.3%), because a reply without the distinguishing detail is classed as uncertain and leaves the belief unchanged.
  • Brief and free-form users degrade performance less than forgetful ones, since "not that one" still excludes a candidate and free-form replies are mapped to the nearest response class.
  • All noisy clarification settings stay above frozen Base without asking (60.5%).
AuthorsOpenReview ↗

Response to Reviewer Fc1x (2/3)

W1 (continued)

c. Users who express their own intentions naturally

Participants in the main human study are told which intent to hold. The table below adds a pilot in which participants choose and express their own intentions, and a simulated user that does the same, comparing ReferClaw with Clarify-or-Answer (Tab. 22):

StudyIntent chosen byRespondentsExamplesIntent resolvedAnswer accuracyClarify-or-Answer accuracy
Main human study (held-out half)assigned10 participants15080.072.057.3
Free-intent pilotparticipant6 participants6076.768.355.0
Self-chosen intent, simulatedsimulated userQwen3-32B1,19781.673.257.8
  • Free-intent pilot (new in the revision): participants watched each clip, asked their own question, answered the clarification, and afterwards marked the referent they had meant. The gap between the methods stays similar to the assigned-intent study (13.3 vs. 14.7 points). With 6 participants and 60 clips, this is preliminary evidence.
  • Simulated self-chosen intent: the simulated user picks an interpretation itself, replies in its own words, and reports its referent afterwards. Answers are scored against that choice.

W2 / Q1. Separation between validation and evaluation

We thank the reviewer for catching this ambiguity in the appendix. In short: yes, they are different sets. The reporting probe and the adapter-promotion probe share no example or video, and both are video-disjoint from the online stream. App. B.7 uses the promotion probe to decide whether to accept model updates, and the 1,000-example held-out set of the reviewer's Table 13 (Tab. 23 in the revised PDF) is the reporting probe. The table below lists the three sets, their sources and sizes, and their use:

SetSourceExamples (ambiguous)VideosUsed for
Online streamBuddyVQA test split6,240 (1,197)308prequential prediction and learning
Promotion probeBuddyVQA validation split, defined within its 704 training videos1,000 (500)120accepting or rolling back adapter updates
Reporting probeother training videos, outside the validation split1,000 (192)70Tab. 23 and Fig. 5 only
  • We verified the separation in the code and logs, and the revision documents it in App. B.7 and the caption of Tab. 23.
  • No probe video is used to train the adapters or the process reward model, and stream inputs are used for learning only after they have been answered.
  • The reporting probe is never used for promotion, rollback, threshold selection, or hyperparameter search, so model selection cannot make the reported learning gains overly optimistic.
AuthorsOpenReview ↗

Response to Reviewer Fc1x (3/3)

Q2. Learning gains with sparse correctness feedback or user corrections

We thank the reviewer for this question on sparse feedback. In short: without gold intervals and boxes, ReferClaw (Full) gains 3.1 points with sparse correctness feedback (verdicts on 20% of inputs) and 4.3 with user corrections on 20% (5.3 and 6.4 on every input).

The table below varies the feedback (full annotations, user corrections, or correctness verdicts) and how sparse it is, i.e., the share of inputs that receive it (Tabs. 8 and 10):

Feedback (no intervals or boxes unless noted)Given onSkillsReferClaw (Full)ReferClaw (Full), probeGain over Base
Full annotated feedback: intent + interval + box + verdict100% of inputs70.174.375.2+8.1
20% of inputs68.971.972.6+5.7
Verdict + one-sentence user correction100% of inputs69.472.673.2+6.4
50% of inputs68.671.672.0+5.4
20% of inputs67.870.570.7+4.3
Binary verdict100% of inputs68.671.572.0+5.3
50% of inputs68.170.671.0+4.4
20% of inputs67.669.369.4+3.1
Strongest adaptation baseline (MetaClaw-style adaptation), full annotated feedback100% of inputs–69.568.6+3.3
Base (frozen, no learning)–66.266.265.80.0

Probe and Gain: as in W1 a. Each value is the mean of three stream orders (std ≤ 0.8). Feedback masks are random and fixed per stream order, so every method receives feedback on the same inputs.

  • Sparse feedback still helps. Verdicts on 20% of inputs give +3.1 points, against +5.3 on every input. Adapter updates start once 128 verified trajectories accumulate, so under the same update rules about a fifth as many updates are performed.
  • Frequency and content both matter. Going from 20% to 100% verdicts adds 2.2 points, while at 20%, replacing verdicts with full annotated feedback adds 2.6.
  • User corrections on every input add 6.4 points without any interval or box, 1.7 below full annotated feedback, because a correction identifies the intended referent and the system grounds it itself.

Q3. Participant assignment and consistency of the human-study gains

We thank the reviewer for this question on the study design. In short: participants were assigned across methods by a counterbalanced Latin square, blind to the system, and the gains are consistent across participants, with at least 9 of 10 favouring ReferClaw on every metric (all 300 examples).

a. How participants were assigned across methods

  • The study has 10 participants, 300 ambiguous examples, and 600 interactions. Each example was answered once under each method, by two different participants.
  • The examples are split into two halves of 150. A calibration half fits the human response model and response-time regression (App. B.2, B.3), and a held-out half is used for all reported estimates and tests.
  • A balanced Latin-square schedule gave each participant 30 examples per method (15 per method in the held-out half).
  • Methods alternated in blocks of five examples, the starting method was counterbalanced across participants, and participants were not told which system generated a question.
  • A method × position term shows no detectable order effect (p = 0.41).

b. Consistency of the gains across participants

The table below reports the held-out results (Tab. 20), three tests of the differences, and the number of participants who favour ReferClaw (App. E.13):

MetricReferClawClarify-or-AnswerDifferenceMixed-effects log-odds [95% CI]McNemar pCluster-bootstrap 95% CIParticipants favouring ReferClaw
Intent resolved (%)80.070.0+10.00.62 [0.28, 0.96]0.012[+3.1, +16.9]9 / 10
Answer accuracy (%)72.057.3+14.70.71 [0.37, 1.05]0.004[+6.5, +22.9]9 / 10
Response time (s)5.16.9−1.8–––10 / 10
Questions rated confusing (%)4.312.7−8.4–––9 / 10

All columns use the 150 held-out examples except the last, which uses all 30 examples per method for each participant.

  • The mixed-effects model has crossed random intercepts for participant and example, McNemar tests pair outcomes on the same example, and the cluster bootstrap resamples participants and examples.
  • ReferClaw's questions are shorter (20.6 vs. 24.0 tokens) and contrast two grounded candidates, which may contribute to the faster responses and fewer confusing questions.
  • Per-participant accuracy differences range from −3.3 to +26.7 points.
AuthorsOpenReview ↗

Official Comment by Authors

Thank you again for your careful review. We hope our responses address your concerns. Learning still gains 5.3 to 6.1 points when users give brief, forgetful, or free-form corrections, and a free-intent pilot shows a margin over Clarify-or-Answer similar to that of the assigned-intent study. We also clarified that the reporting and promotion probes are disjoint, reported results with sparse feedback, and detailed the counterbalanced assignment and per-participant consistency of the human study. We are grateful for your time and for your recognition of the problem we address and of our controlled analysis.

Reviewer_dtJyOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces ReferClaw, a framework for resolving ambiguous user intent in egocentric video question answering. Three cooperating agents construct spatiotemporally grounded interpretations, select clarification questions by balancing information gain against respondent cost, and generate answers from the selected interpretation and evidence. The framework also incorporates intent-specific skills and agent-level updates to learn from verified interaction feedback. Experiments on augmented egocentric QA benchmarks, together with transfer evaluations and component ablations, show improvements in answer accuracy, grounding, and clarification efficiency.

Critical strengths
  1. Representing competing interpretations with explicit temporal intervals and object tracks provides a concrete mechanism for resolving references across repeated encounters in egocentric video.
  2. The evaluation separates clarification from online adaptation and includes matched asking-disabled controls, component ablations, temporal-ambiguity diagnostics, and transfer tests.
Critical weaknesses
  1. The online-learning results depend on substantially richer feedback than ordinary clarification exchanges provide: after each prediction, the system receives the resolved intent, supporting temporal intervals and spatial boxes, and a correctness verdict. These annotations directly support failure attribution and policy updates, but the main text does not explain how they would be obtained during actual assistance or account for the effort required to produce them. Although adaptation baselines receive the same feedback, making the comparison fair, the results primarily demonstrate learning under dense supervision; they do not establish whether the gains remain achievable with feedback that users can realistically provide.
  2. The clarification gate in Equation (3) suppresses questions whenever candidate interpretations imply the same answer, even if the intended referent remains unresolved. This conflicts with the grounding component of Equation (1): two encounters can support the same textual answer while requiring different temporal intervals or object tracks. EFA then selects one interpretation, and its verifier checks consistency with that selection rather than establishing that it matches the user's intent. Consequently, the system can produce a supported answer with incorrect target evidence. The paired-intent experiment uses mutually exclusive target answers, so it does not test this limitation of the central clarification mechanism.
Questions for authors

How frequently do distinct candidate intents share the same answer but require different evidence, and what happens to grounding accuracy and respondent cost when the clarification gate also considers these cases?

5 comments
AuthorsOpenReview ↗

Response to Reviewer dtJy (1/3)

We sincerely thank the reviewer for the valuable time spent on our paper and the thoughtful feedback. We are encouraged that the reviewer likes our spatio-temporally grounded intent representation and controlled experimental validation. We address the major concerns below and also reflect them in the revision (blue). Table, section, and appendix numbers refer to the PDF.

W1. Feedback substantially richer than ordinary clarification exchanges

We thank the reviewer for raising the practicality of the feedback. In short: with feedback that users can realistically provide (binary verdicts), ReferClaw (Full) still retains 5.3 of its 8.1-point gain.

a. What each annotation contributes

To measure what each annotation contributes, we remove the annotations of the default setting (S0) one at a time under the same stream and update rules (new in App. D.3). The table below lists, for each setting, who provides the feedback, together with the accuracy, Acc@GQA, and attribution agreement of ReferClaw (Full) (Tab. 8):

SettingRevealed after each predictionWho provides itReferClaw (Full)ReferClaw (Full), Acc@GQAAttribution agreementGain over Base
Base (frozen)––66.236.2–0.0
S0 (default)intent + interval + box + verdictannotator, offline74.341.891%+8.1
S1intent + interval + verdictannotator, offline74.041.290%+7.8
S2intent + verdictannotator or user73.440.687%+7.2
S3verdictuser: one binary verdict71.539.683%+5.3
S4clarification exchange onlyuser, during the interaction68.938.078%+2.7

Gain: accuracy gain over frozen Base (66.2%), in points. Attribution agreement: agreement of automatic failure attribution with adjudicated labels from two authors, used only as an offline audit (App. B.5).

Along this removal order, intervals and boxes add 0.9 points of accuracy (95% CI [+0.3, +1.5]), the intent 1.9, the verdict 2.6, and the clarification exchange 2.7. These are differences along one order, not independent contributions. The clarification exchange and the verdict, both of which a user can give, account for 5.3 of the 8.1 points, while intervals, boxes, and the intent matter more for grounding and attribution:

  • The clarification exchange alone (S4) adds 2.7 points, since on asked inputs (18.6%) the candidate selected by the user's reply serves as a self-label, matching the target intent on 81% of them.
  • Intervals and boxes add little accuracy because the system grounds the identified referent itself, but they raise Acc@GQA (40.6 → 41.8) and attribution agreement (87% → 91%).
  • The intent adds more because the attribution rules need it to send each failure to the right agent.

b. How the feedback would be obtained during actual assistance

During actual assistance, the revised paper (Sec. 4.4, App. D.3) specifies that:

  • The learner uses only what the user gives, namely the clarification reply, a verdict when the user can give one, and a one-sentence correction when volunteered (at most 15 tokens, 8.6 on average).
  • Annotated intervals and boxes are never requested from users and serve only for offline auditing.
AuthorsOpenReview ↗

Response to Reviewer dtJy (2/3)

W1 (continued)

c. Whether the gains remain achievable with feedback users can realistically provide

The table below compares ReferClaw (Full) trained with different (simulated) user feedback against frozen Base and the strongest adaptation baseline, in the setup the reviewer found fair (Tabs. 8 and 10).

Learner and feedbackAccuracyProbeGain over Base
Base (frozen), no learning66.265.80.0
Strongest adaptation baseline (MetaClaw-style adaptation), full annotated feedback69.568.6+3.3
ReferClaw (Full), full annotated feedback (S0)74.375.2+8.1
ReferClaw (Full), clarification exchange only68.968.8+2.7
ReferClaw (Full), verdict71.572.0+5.3
ReferClaw (Full), verdict, 20% flipped70.671.0+4.4
ReferClaw (Full), verdict + one-sentence correction72.673.2+6.4
ReferClaw (Full), composite realistic user70.370.6+4.1

Composite: feedback on a random 50% of interactions (half verdicts, half verdicts with a correction), with 50% of utterances (clarification replies and corrections) brief, forgetful, partially correct, or free-form as defined in App. D.3, and 10% of verdicts flipped. Probe: accuracy on a separate held-out reporting probe of 1,000 examples. Means over three runs (std ≤ 0.8). Tab. 9 gives accuracy by question type.

  • With verdicts alone, ReferClaw (Full) reaches 71.5% (+5.3, 95% CI [+4.2, +6.4]), above the strongest adaptation baseline with full annotated feedback (69.5%). The baseline was evaluated only with this feedback, so this is not a matched comparison. The held-out probe shows the same ordering (72.0% vs. 68.6%).
  • Verdicts suffice because learning stays intent-specific. Without the intent, failures are still attributed to an agent from the logged trajectory, and on inputs where the assistant asked, the candidate selected by the user's reply stands in for the annotated intent (App. D.3). In contrast, the baseline's generic skills store no candidate contrast, and its final-answer reward updates all adapters, so it makes little use even of full annotated feedback.
  • A one-sentence correction adds 1.1 points over verdicts alone (+6.4, 1.7 below full annotated feedback), mainly on inputs where the assistant did not ask, because there a correction names the referent and a verdict does not.
  • Imperfect feedback reduces the gain only moderately. Flipping 20% of the verdicts still leaves +4.4 points. The composite realistic user, which combines feedback on half of the interactions, noisy utterances, and flipped verdicts, still reaches 70.3% (+4.1), above the strongest adaptation baseline with full annotated feedback.

W2 / Q. Candidate interpretations with the same answer but different evidence

We appreciate the insightful comments. In short: such cases account for about 5.0% of test inputs, and when the gate also considers them, their grounding accuracy (Acc@GQA) rises from 29.0 to 40.7 at 0.009 extra respondent cost per input.

a. How frequently distinct candidate intents share the same answer (Q)

We count, among the 6,240 test inputs, those on which distinct candidate intents share the same answer but require different evidence (App. E.7):

CountShare
Candidate-bearing inputs (frozen Base returns ≥ 2 candidates)2,99548.0% of 6,240 test inputs
Same answer, different evidence (detected)3125.0% of the 6,240 test inputs
of which box-annotated24177.2% of the subset
by type: Recall / Recognition / Advisory / Scrutinization118 / 87 / 49 / 58
  • An input is counted when at least two frozen-Base candidates with prior mass ≥ 0.1 have judge-equivalent provisional answers but different evidence (overlap < 0.5 or different tracks).
  • Two annotators audited 100 random subset inputs and confirmed 94% of them (κ = 0.81).
  • Across mass thresholds {0.05, 0.1, 0.2} and overlap thresholds {0.3, 0.5, 0.7}, the subset size stays within 247–371.

Distinct candidate intents with the same answer thus do occur on Augmented BuddyVQA, on about 5% of test inputs (4.0–5.9% across detection thresholds).

AuthorsOpenReview ↗

Response to Reviewer dtJy (3/3)

W2 / Q (continued)

b. Conflict with the grounding component of Eq. 1

The gate of Eq. 3 was designed around answer utility, so it stays closed when the candidates share an answer, even though a question could still raise the grounding term λgUgrd\lambda_g U_{\mathrm{grd}} of Eq. 1 when the two encounters require different evidence. The revision therefore adds a grounding-aware gate (Sec. 3.2, App. E.7), which also opens when the expected evidence mismatch of answering from the top candidate i⋆i^\star, Gt=1−∑jpt(zj) o(ei⋆,ej),o=min⁡(tIoU,sIoU),G_t = 1-\sum_j p_t(z_j)\,o(e_{i^\star},e_j),\qquad o=\min(\mathrm{tIoU},\mathrm{sIoU}), exceeds τa\tau_a, and asks only if λg Gt IGt(c)/H(pt)>λC Cx(c)\lambda_g\,G_t\,\mathrm{IG}_t(c)/H(p_t) > \lambda_C\,C_x(c), i.e., if the expected reduction of this mismatch outweighs the respondent cost. It reuses τa\tau_a and λg=0.5\lambda_g=0.5 from the validation split and is enabled when the assistant shows or acts on its evidence (e.g., replays the clip), while the main results use the current gate.

c. Grounding accuracy and respondent cost when the gate also considers these cases (Q)

The table below compares ReferClaw (Full) with its current gate and with the grounding-aware gate at λg=0.5\lambda_g=0.5 and 11 (Tab. 16), on the detected subset of W2 a, on 120 same-answer pairs, and on all test inputs. Each pair asks the same question twice, with two intended encounters that have the same answer but disjoint evidence, so only correct grounding tells them apart. Model, history, and seed are reset between the two presentations. A pair counts as correct when both presentations are answered correctly with T-IoU and S-IoU ≥ 0.5 against their own target. Respondent cost Cx∈[0,1]C_x \in [0,1] is the normalized cost of a question, combining effort, response time, and failure risk (App. B.2), averaged per input with 0 when no question is asked.

PopulationReferClaw (Full), current gateGrounding-aware, λg=0.5\lambda_g=0.5Grounding-aware, λg=1\lambda_g=1
Ask rate312 subset4.5%46.2%68.3%
Turns per input312 subset0.050.520.79
T-mIoU312 subset41.355.859.4
S-mIoU241 boxed subset33.646.949.8
Acc@GQA241 boxed subset29.040.743.6
Same-answer pairs, both grounded120 pairs23.3%61.7%68.3%
Answer accuracyall 6,24074.374.474.3
Acc@GQA3,914 boxed41.842.542.7
Ask-P / Ask-R1,197 ambiguous85.8 / 83.185.6 / 83.185.1 / 83.1
Unnecessary-ask rate5,043 well-specified3.9%6.6%8.1%
Respondent cost per inputall 6,2400.0780.0870.092

Population: the 6,240 test inputs comprise 1,197 intent-ambiguous and 5,043 well-specified questions (App. C.1).

  • Large subset gain, small overall gain. At λg=0.5\lambda_g=0.5, subset Acc@GQA rises by 11.7 points (95% CI [+8.1, +15.3]). Since the 241 inputs are 6.2% of the 3,914 on which overall Acc@GQA is computed, the overall gain is about 0.7.
  • Answer accuracy remains essentially unchanged (74.3% vs. 74.4%), as does asking on intent-ambiguous inputs.
  • Both-grounded pairs rise from 23.3% to 61.7%. Without a question, both presentations receive identical inputs, so at most one can be grounded. The current gate still asks on some pairs because it compares the system's provisional answers, which can differ even when the reference answers agree.
  • Cost. The gate asks on about 130 more subset inputs (147 more turns), and the extra 0.009 per input amounts to 0.43 per additional asked input, close to the current 0.42. These questions also account for the rise in the answer-only unnecessary-ask rate (130 / 5,043 = 2.6 vs. 2.7 points), which ignores grounding needs. λg=1\lambda_g=1 asks on 22.1 more points of the subset for only +2.9 Acc@GQA.
Reviewer_iZHCOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces ReferClaw, a multi agent framework for egocentric video QA that explicitly represents multiple grounded user-intent hypotheses and asks clarification questions when the intent is unclear. The framework includes intent candidate construction, cost-aware clarification based on expected information gain, evidence-fused answering, and interaction-based adaptation through retrieved skills and agent-specific GRPO updates. Experiments on diverse benchmarks show that ReferClaw improves both answer accuracy and grounded answering over direct and interactive baselines, while ablations and transfer analyses support the contributions of its individual components.

Critical strengths

S1. Grounded intent representation.

A key strength is the paper’s explicit formulation of ambiguous user intent as multiple visually grounded hypotheses rather than forcing the model to commit to a single interpretation. By associating each candidate interpretation with temporal and spatial evidence, the framework directly addresses referential ambiguity that naturally arises from repeated encounters and similar objects in egocentric videos. I find this formulation both well motivated and consequential, as it provides a clear foundation for deciding when clarification is necessary and what the clarification should distinguish.

S2. Careful validation of the value of clarification.

The paper provides strong empirical support that the gains come from selective clarification rather than simply adding more interaction or model components. In particular, the cost-aware ablations and state-matched asking-disabled controls isolate the immediate benefit of asking while also showing that the ambiguity gate, information-gain criterion, and stopping mechanism help avoid unnecessary user burden. These experiments substantially strengthen the paper because they directly validate the central claim that targeted clarification can improve intent-aligned answering at a reasonable interaction cost.

Critical weaknesses

1. The interaction-learning setting relies on supervision that is substantially stronger than what would normally be available from real users.

The current prediction itself does not receive any gold information, so there is no direct test-time leakage. However, after each interaction, the adapting variants are given the annotated target intent, its resolved question, the corresponding gold evidence, and the correctness verdict. ReferClaw Skills can immediately use this verified failure information to synthesize a new intent-specific skill, while ReferClaw Full additionally uses these trajectories for agent-attributed GRPO updates that affect subsequent interactions. Thus, the reported online improvement is not obtained from ordinary user feedback alone, but from a fairly rich post-hoc supervision signal that explicitly identifies what the intended interpretation and supporting visual evidence should have been.

This distinction is important because the paper frames the method as learning from interaction, whereas a deployed assistant is unlikely to receive a resolved intent and annotated visual evidence after every mistake. A real user may only say that the answer is wrong, provide an incomplete correction, or answer a clarification question without supplying the temporal interval or spatial evidence needed to attribute the failure to IGA, CCA, or EFA. I therefore find it unclear how much of the gain from Skills and Full would remain under realistically weak and noisy feedback. An evaluation in which adaptation is restricted to signals that an ordinary user could plausibly provide, such as clarification responses and binary or corrective feedback without gold evidence, would be important for establishing the practical significance of the interaction-learning contribution.

2. The paper does not fully establish that the complexity of the full multi-agent architecture is necessary for the observed gains.

ReferClaw combines a temporal proposal model, Grounding DINO, SAM2 tracking, three agent-specific LoRA adapters, explicit candidate beliefs and Bayesian updates, information-gain and cost estimation, a deterministic interaction controller, an answer verifier, skill retrieval, and GRPO-based adaptation. The paper provides many subtractive ablations showing that removing individual components hurts performance, but Appendix E.5 also reports a much simpler control, "Single agent, same structured steps, one adapter," which reaches 71.6 percent answer accuracy and 39.9 percent Acc@GQA, compared with 74.3 percent and 41.8 percent for the full system. Its interaction cost is also close to Full, with cost values of 0.44 versus 0.42 and turn counts of 0.50 versus 0.46 per candidate-bearing input. This suggests that much of the benefit may already come from the structured grounded-candidate and clarification procedure rather than from the three-agent decomposition itself.

Consequently, the existing ablations demonstrate that the components are useful within the proposed architecture, but they do not cleanly show that this architecture is the most appropriate level of complexity for the task. In particular, most experiments start from Full and remove pieces, rather than starting from a strong simpler system and incrementally adding the multi-agent machinery. A stronger constructive baseline, for example a unified model using the same grounded candidate representation, clarification objective, and evidence selection but without separate agent-specific adapters and cross-agent communication, would make it much clearer which gains genuinely require the proposed multi-agent design. Without such a comparison, I find it difficult to assess whether the additional architectural and implementation complexity is justified by the relatively modest improvement over the single-agent structured control.

Questions for authors

Q1. Your interaction-learning setting reveals the annotated target intent, resolved question, supporting evidence, and correctness verdict after each interaction. How much of the improvement from ReferClaw Skills and ReferClaw Full remains if adaptation is restricted to feedback that a real user could plausibly provide, such as the clarification response and a binary or textual correction, without access to annotated temporal or spatial evidence?

Q2. Appendix E.5 reports that the single-agent control with the same structured steps reaches 71.6 percent answer accuracy and 39.9 percent Acc@GQA, compared with 74.3 percent and 41.8 percent for the full system. Can the authors clarify which aspect of the three-agent decomposition accounts for this remaining gap, and whether the gain comes primarily from role-specific adapters, separate prompts, cross-agent communication, or the controller?

4 comments
AuthorsOpenReview ↗

Response to Reviewer iZHC (1/3)

We sincerely thank the reviewer for the valuable time spent on our paper and the constructive comments. We are glad that the reviewer likes our grounded intent representation and careful experimental validation. We address the major concerns below and also reflect them in the revision (blue). Table, section, and appendix numbers refer to the PDF.

W1 / Q1. Supervision substantially stronger than what real users would provide

We thank the reviewer for this important point on feedback realism. In short: restricted to using feedback a real user could plausibly provide, ReferClaw (Full) keeps 5.3 of its 8.1-point gain with binary verdicts and 6.4 with textual corrections.

a. How much of the improvement remains under weaker post-hoc supervision

To measure how much of the improvement from ReferClaw Skills and ReferClaw Full remains, we remove the box, interval, intent, and verdict in turn (S1–S4, new in App. D.3), from the most to the least costly, under the same stream and update rules. In each setting, gold labels enter learning only through what that setting reveals. Attribution and rewards use the revealed intent with the system's own evidence (S1, S2) or the most likely candidate after clarification (S3, S4). GRPO rollouts are completed by the assistant's own model of the user instead of the simulator, and the process reward model is retrained per setting. The table below (Tab. 8) reports, for each setting, the accuracy of Skills and ReferClaw (Full), Full's Acc@GQA, and attribution agreement. Gain: accuracy gain over frozen Base (66.2%), in points.

SettingRevealed after each predictionSkillsReferClaw (Full)ReferClaw (Full), probeReferClaw (Full), Acc@GQAAttribution agreementGain, SkillsGain, ReferClaw (Full)
Base (frozen)–66.266.265.836.2–0.00.0
S0 (default)intent + interval + box + verdict70.174.375.241.891%+3.9+8.1
S1intent + interval + verdict70.074.074.941.290%+3.8+7.8
S2intent + verdict69.873.474.140.687%+3.6+7.2
S3verdict68.671.572.039.683%+2.4+5.3
S4clarification reply only67.568.968.838.078%+1.3+2.7

Attribution agreement: agreement of automatic failure attribution with adjudicated labels from two authors, used only as an offline audit (App. B.5). Probe: accuracy on a held-out reporting probe of 1,000 examples. Tab. 9 gives accuracy by question type.

Along this removal order, intervals and boxes add 0.9 points (95% CI [+0.3, +1.5]), the intent 1.9, the verdict 2.6, and the clarification reply 2.7. These are differences along one order, not independent contributions. Lower-burden feedback, a clarification reply when asked plus a verdict on every input, retains 5.3 of the 8.1 points.

  • Intervals and boxes add little answer accuracy because the system grounds the identified referent itself, but they raise Acc@GQA (40.6 → 41.8) and attribution agreement (87% → 91%).
  • The intent adds more because the attribution rules need it. Without it, agreement falls from 87% to 83%.
  • The clarification reply alone (S4) still adds 2.7 points, as the candidate selected by the reply matches the target intent on 81% of asked inputs.
AuthorsOpenReview ↗

Response to Reviewer iZHC (2/3)

W1 / Q1 (continued)

b. Realistically weak and noisy feedback (simulated)

We restrict adaptation to what a real user may provide, namely answering a clarification question, saying whether the answer is correct (a binary verdict), or giving a correction that may be only partially correct, each in clean and noisy form. The table below (Tabs. 10 and 11) reports the learning results against frozen Base and the strongest adaptation baseline. Noisy replies and corrections are simulated from the target intent. A partially correct correction names one correct and one incorrect attribute, and a noisy utterance (clarification reply or correction) is brief (at most three words), forgetful (missing the distinguishing detail), partially correct, or free-form (in the user's own words).

User feedbackConditionSkillsReferClaw (Full)ReferClaw (Full), probeGain, SkillsGain, ReferClaw (Full)
Clarification reply onlyclean67.568.968.8+1.3+2.7
50% of replies noisy67.168.268.1+0.9+2.0
Binary verdictclean68.671.572.0+2.4+5.3
10% of verdicts flipped68.371.171.5+2.1+4.9
20% of verdicts flipped68.070.671.0+1.8+4.4
Verdict + one-sentence correctionclean69.472.673.2+3.2+6.4
50% of corrections partially correct68.971.772.2+2.7+5.5
30% of corrections vague or missing69.071.872.3+2.8+5.6
20% of verdicts flipped68.871.872.3+2.6+5.6
Composite realistic usersee below67.970.370.6+1.7+4.1
Strongest adaptation baseline (MetaClaw-style adaptation)full annotated feedback–69.568.6–+3.3

Composite: feedback on a random 50% of interactions (half verdicts, half verdicts with a correction), with 50% of utterances noisy and 10% of verdicts flipped. Means over three runs (std ≤ 0.8).

  • With binary verdicts alone, ReferClaw (Full) gains 5.3 points (95% CI [+4.2, +6.4]) and Skills 2.4. The strongest adaptation baseline gains 3.3 with full annotated feedback, the only feedback it was evaluated with, so this is not a matched comparison. The held-out probe shows the same ordering (72.0% vs. 68.6%).
  • Failures can still be attributed to IGA, CCA, or EFA without gold evidence. With verdicts alone, attribution from the logged trajectory agrees with adjudicated labels on 83% of failures (Tab. 8), and on asked inputs the candidate selected by the user's reply stands in for the annotated intent (App. D.3).
  • A one-sentence correction raises the gain to 6.4 points for ReferClaw (Full) and 3.2 for Skills (1.7 and 0.7 below full annotated feedback), mainly on inputs where the assistant did not ask, because there a correction names the referent and a verdict does not.
  • Noisy feedback reduces the gain only moderately. With 20% of the verdicts flipped, ReferClaw (Full) still gains 4.4 points, and with half of the corrections partially correct, 5.5. The composite realistic user, which combines feedback on half of the interactions, noisy utterances, and flipped verdicts, still yields +4.1 for ReferClaw (Full), above the strongest adaptation baseline, and +1.7 for Skills.

c. Practical significance of the interaction-learning contribution

  • In practice, the learner would use clarification replies, verdicts when users can give them, and volunteered corrections. Annotated intervals and boxes serve only for offline auditing.
  • These settings affect only the learning contribution, since the clarification gains are measured on frozen systems (66.2% vs. 60.5% for Base with and without asking).
AuthorsOpenReview ↗

Response to Reviewer iZHC (3/3)

W2 / Q2. Whether the multi-agent architecture is necessary for the observed gains

We thank the reviewer for suggesting the constructive comparison. In short: the single-agent structured control captures most of the benefit, and the remaining gap comes mainly from agent-attributed updates (2.0 of 2.7 points), not from role-specific adapters or separate prompts.

a. A stronger constructive baseline

We agree that the single-agent structured control captures most of the benefit, and we read this as support for our core design. This control (A0, the row the reviewer cites in App. E.5) differs from ReferClaw (Full) only in running the three roles as one prompt sequence on one adapter. As the reviewer suggests, the revision starts from this simpler system and adds the multi-agent components one at a time. The table below reports, for each step, accuracy without and with learning and Acc@GQA (Sec. 4.3, App. E.6, Tab. 15):

ConfigurationFrozenLearnedAcc@GQATrainableGPU s / inputUpdate GPU-h / 1k
A0: single agent, same structured steps, one adapter64.171.639.91 adapter4.82.5
A0′: A0 with a rank-48 adapter (capacity-matched)64.371.940.11 (rank 48)4.82.5
A1: + role-specific prompts, one shared adapter65.772.040.21 adapter4.82.5
A2: + one adapter per agent, all updated66.272.339.43 adapters5.03.0
A3: + agent-attributed updates = ReferClaw (Full)66.274.341.83 adapters5.02.8
Skills (A3 without adapter updates)66.270.138.9none5.00.3

A0′ raises the adapter rank from 16 to 48 to match the trainable parameters of the three adapters in A2 and A3.

b. Which aspect accounts for the remaining gap (Q2)

  • Agent-attributed updates (A3): +2.0 (72.3% → 74.3%), the largest step. Updating all adapters on every failure (A2) lowers Acc@GQA from 40.2 to 39.4, and attributed updates raise it to 41.8, consistent with reduced interference between agents.
  • Separate, role-specific prompts (A1): +0.4 (+1.6 before learning).
  • Role-specific adapters, one per agent (A2), updated together: +0.3, the same as extra capacity alone (A0′), and their benefit appears together with attributed updates.
  • Cross-agent communication and the controller are shared by all rows, so this comparison does not estimate their contribution.

c. Whether the additional complexity is justified

  • The gain concentrates on clarification and grounding. ReferClaw (Full) improves over the single-agent structured control by 2.7 points of accuracy and 1.9 of Acc@GQA (Tab. 15), spread evenly over the four question types (+2.6 to +2.9 points). Compared with updating all adapters on every failure, attributed updates raise clarification success from 78.3% to 81.4% and Acc@GQA from 39.4 to 41.8, while asking less often (38.7% vs. 42.0% of candidate-bearing inputs) at lower cost per asked input (0.42 vs. 0.46, Tab. 17).
  • Attributed updates in the multi-agent design leave correctly acting agents untouched. With separate agents, each failure updates only the agent responsible for it, so correcting one role does not change the behaviour of the others. For example, when a failure comes from a question that does not separate the candidates, only CCA is updated, while the grounding of IGA and the answers of EFA, which acted correctly, stay unchanged.
  • The separate roles pay off mainly through attributed learning. About 2.1 points already appear before learning, mostly from role-specific prompts (+1.6). During learning, three adapters updated together learn less than the single adapter (+6.1 vs. +7.5 points), whereas attributed updates learn the most (+8.1).

The added complexity is therefore justified for an assistant that keeps learning from its interactions, since only the multi-agent design lets each failure be traced to grounding, asking, or answering and corrected in that agent alone.

AuthorsOpenReview ↗

Official Comment by Authors

Thank you again for your thoughtful review. We hope our responses address your two main concerns. With feedback that ordinary users can give, ReferClaw (Full) keeps 5.3 of its 8.1-point learning gain, and the constructive comparison you suggested shows that agent-attributed updates provide the largest step beyond the single-agent control, mainly by improving clarification and grounding. Both analyses are now included in the revision. We sincerely appreciate your time and your positive assessment of our grounded intent representation and of our validation of selective clarification.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Responses

We thank the AC and all reviewers for their time and constructive comments, and we took each comment very seriously. As several of your comments could only be answered with extensive and rigorous new experiments, we chose to complete these experiments before posting our responses, and summarize them below. With less than a day left in the discussion period, we would appreciate knowing whether our responses resolve your concerns, or whether any point remains that we can still clarify.

The reviewers found that representing each candidate interpretation with its own temporal and spatial evidence is a well-motivated and concrete mechanism for resolving references across repeated encounters (iZHC, dtJy), that the method targets a real limitation of current video QA systems, namely answers that match the video but refer to the wrong encounter (Fc1x), and that the controls convincingly isolate the benefit of selective clarification while accounting for user effort (iZHC, dtJy, Fc1x).

Following the reviewers' suggestions, we ran new sets of experiments. Together, they show that ReferClaw's gains stem from grounding each candidate interpretation in its own visual evidence and from correcting each failure in the agent responsible for it, and that these gains persist under the more realistic and demanding conditions the reviewers raised.

  • Learning with realistic feedback (iZHC W1/Q1; dtJy W1; Fc1x W1, Q2). We remove the annotations one at a time and then simulate user feedback. With binary verdicts alone, ReferClaw (Full) keeps 5.3 of its 8.1-point gain and still exceeds the strongest adaptation baseline given full annotated feedback (71.5% vs. 69.5%). A one-sentence correction raises the gain to 6.4 points, and brief, forgetful, or free-form corrections still give 5.3 to 6.1. Flipping 20% of the verdicts leaves 4.4 points, and verdicts or corrections on only 20% of inputs give 3.1 and 4.3. Intervals and boxes, the costliest annotations, add only 0.9 points, so most of the learning gain does not require dense supervision.
  • Multi-agent design versus a single-agent control (iZHC W2/Q2). Starting from the single-agent structured control, we add role-specific prompts, separate adapters, and agent-attributed updates one at a time. The full system adds 2.7 points of accuracy and 1.9 of Acc@GQA, and attributed updates give the largest step (+2.0). Compared with updating all adapters, they raise clarification success from 78.3% to 81.4% and Acc@GQA from 39.4 to 41.8 while asking less often, because each failure changes only the responsible agent.
  • Same answer, different evidence (dtJy W2/Q). Such inputs account for 5.0% of the test set. A grounding-aware gate, which also asks when the candidates point to different evidence, raises Acc@GQA on them from 29.0 to 40.7 and the share of same-answer pairs grounded in both presentations from 23.3% to 61.7%, at 0.009 extra respondent cost per input and without changing answer accuracy.

Responses to other questions

  • How feedback is obtained in deployment (dtJy W1). The learner uses only clarification replies, verdicts when available, and volunteered corrections. Annotated intervals and boxes are never requested from users and serve only for offline auditing.
  • Users who choose their own intents (Fc1x W1). In a free-intent pilot, participants asked their own questions and marked their referent afterwards. The margin over Clarify-or-Answer stays close to that of the assigned-intent study (13.3 vs. 14.7 points), and a simulated user that picks its own interpretation shows a similar gap.
  • Human study design (Fc1x Q3). Methods were assigned by a counterbalanced Latin square, and participants were blind to the system. The gains are significant (McNemar p = 0.012 and 0.004) and shared across participants, as 9 of 10 resolve more intents and answer more accurately with ReferClaw and all 10 respond faster.
  • Validation and evaluation sets (Fc1x W2/Q1). The reporting probe and the adapter-promotion probe share no example or video, and both are video-disjoint from the online stream.

All new results are included in the revised PDF and marked in blue.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.