- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
ReferClaw helps an egocentric video assistant clarify ambiguous requests before answering. Three agents identify possible interpretations, choose a short clarification question, and answer from the selected evidence. The system also learns reusable skills and updates agent-specific adapters from feedback. Experiments cover augmented video QA datasets, image-based transfer, and human respondents.
Critical strengths
-
The method addresses a real failure of video QA which is an answer can match the video but refer to the wrong encounter. Sharing grounded candidates between clarification and answering is a sensible way to address this.
-
The controls are convincing. Frozen Base reaches 66.2% accuracy versus 60.5% with asking disabled at the same entering state. The paired-intent test also shows that answers follow the user's clarification rather than a fixed interpretation.
-
The paper considers user effort as well as accuracy. It includes short-question budgets, unnecessary-asking measurements, transfer tests, and a human comparison in which ReferClaw's questions are answered faster and cause less confusion.
Critical weaknesses
-
The evaluation relies on fairly ideal conditions. Learning receives the correct intent and annotated evidence after every prediction, while participants in the human study are told which intent to hold. These results are useful, but it remains unclear how well the system works when users give brief corrections, forget details, or express their own intentions naturally.
-
The separation between validation and evaluation needs clarification. Appendix B.7 uses 1000 examples to decide whether to accept model updates, and Table 13 reports results on a 1000-example held-out set. Are these different sets? If the same examples are used for both purposes, the reported learning gains may be overly optimistic.
Questions for authors
-
Are the reporting probe and adapter-promotion probe separate, and are both video-disjoint from the online stream?
-
What learning gains remain with sparse correctness feedback or user corrections, without gold intervals and boxes?
-
How were participants assigned across methods, and how consistent are the human-study gains across participants?
4 comments
Response to Reviewer Fc1x (2/3)
W1 (continued)
c. Users who express their own intentions naturally
Participants in the main human study are told which intent to hold. The table below adds a pilot in which participants choose and express their own intentions, and a simulated user that does the same, comparing ReferClaw with Clarify-or-Answer (Tab. 22):
| Study | Intent chosen by | Respondents | Examples | Intent resolved | Answer accuracy | Clarify-or-Answer accuracy |
|---|---|---|---|---|---|---|
| Main human study (held-out half) | assigned | 10 participants | 150 | 80.0 | 72.0 | 57.3 |
| Free-intent pilot | participant | 6 participants | 60 | 76.7 | 68.3 | 55.0 |
| Self-chosen intent, simulated | simulated user | Qwen3-32B | 1,197 | 81.6 | 73.2 | 57.8 |
- Free-intent pilot (new in the revision): participants watched each clip, asked their own question, answered the clarification, and afterwards marked the referent they had meant. The gap between the methods stays similar to the assigned-intent study (13.3 vs. 14.7 points). With 6 participants and 60 clips, this is preliminary evidence.
- Simulated self-chosen intent: the simulated user picks an interpretation itself, replies in its own words, and reports its referent afterwards. Answers are scored against that choice.
W2 / Q1. Separation between validation and evaluation
We thank the reviewer for catching this ambiguity in the appendix. In short: yes, they are different sets. The reporting probe and the adapter-promotion probe share no example or video, and both are video-disjoint from the online stream. App. B.7 uses the promotion probe to decide whether to accept model updates, and the 1,000-example held-out set of the reviewer's Table 13 (Tab. 23 in the revised PDF) is the reporting probe. The table below lists the three sets, their sources and sizes, and their use:
| Set | Source | Examples (ambiguous) | Videos | Used for |
|---|---|---|---|---|
| Online stream | BuddyVQA test split | 6,240 (1,197) | 308 | prequential prediction and learning |
| Promotion probe | BuddyVQA validation split, defined within its 704 training videos | 1,000 (500) | 120 | accepting or rolling back adapter updates |
| Reporting probe | other training videos, outside the validation split | 1,000 (192) | 70 | Tab. 23 and Fig. 5 only |
- We verified the separation in the code and logs, and the revision documents it in App. B.7 and the caption of Tab. 23.
- No probe video is used to train the adapters or the process reward model, and stream inputs are used for learning only after they have been answered.
- The reporting probe is never used for promotion, rollback, threshold selection, or hyperparameter search, so model selection cannot make the reported learning gains overly optimistic.
Response to Reviewer Fc1x (3/3)
Q2. Learning gains with sparse correctness feedback or user corrections
We thank the reviewer for this question on sparse feedback. In short: without gold intervals and boxes, ReferClaw (Full) gains 3.1 points with sparse correctness feedback (verdicts on 20% of inputs) and 4.3 with user corrections on 20% (5.3 and 6.4 on every input).
The table below varies the feedback (full annotations, user corrections, or correctness verdicts) and how sparse it is, i.e., the share of inputs that receive it (Tabs. 8 and 10):
| Feedback (no intervals or boxes unless noted) | Given on | Skills | ReferClaw (Full) | ReferClaw (Full), probe | Gain over Base |
|---|---|---|---|---|---|
| Full annotated feedback: intent + interval + box + verdict | 100% of inputs | 70.1 | 74.3 | 75.2 | +8.1 |
| 20% of inputs | 68.9 | 71.9 | 72.6 | +5.7 | |
| Verdict + one-sentence user correction | 100% of inputs | 69.4 | 72.6 | 73.2 | +6.4 |
| 50% of inputs | 68.6 | 71.6 | 72.0 | +5.4 | |
| 20% of inputs | 67.8 | 70.5 | 70.7 | +4.3 | |
| Binary verdict | 100% of inputs | 68.6 | 71.5 | 72.0 | +5.3 |
| 50% of inputs | 68.1 | 70.6 | 71.0 | +4.4 | |
| 20% of inputs | 67.6 | 69.3 | 69.4 | +3.1 | |
| Strongest adaptation baseline (MetaClaw-style adaptation), full annotated feedback | 100% of inputs | – | 69.5 | 68.6 | +3.3 |
| Base (frozen, no learning) | – | 66.2 | 66.2 | 65.8 | 0.0 |
Probe and Gain: as in W1 a. Each value is the mean of three stream orders (std ≤ 0.8). Feedback masks are random and fixed per stream order, so every method receives feedback on the same inputs.
- Sparse feedback still helps. Verdicts on 20% of inputs give +3.1 points, against +5.3 on every input. Adapter updates start once 128 verified trajectories accumulate, so under the same update rules about a fifth as many updates are performed.
- Frequency and content both matter. Going from 20% to 100% verdicts adds 2.2 points, while at 20%, replacing verdicts with full annotated feedback adds 2.6.
- User corrections on every input add 6.4 points without any interval or box, 1.7 below full annotated feedback, because a correction identifies the intended referent and the system grounds it itself.
Q3. Participant assignment and consistency of the human-study gains
We thank the reviewer for this question on the study design. In short: participants were assigned across methods by a counterbalanced Latin square, blind to the system, and the gains are consistent across participants, with at least 9 of 10 favouring ReferClaw on every metric (all 300 examples).
a. How participants were assigned across methods
- The study has 10 participants, 300 ambiguous examples, and 600 interactions. Each example was answered once under each method, by two different participants.
- The examples are split into two halves of 150. A calibration half fits the human response model and response-time regression (App. B.2, B.3), and a held-out half is used for all reported estimates and tests.
- A balanced Latin-square schedule gave each participant 30 examples per method (15 per method in the held-out half).
- Methods alternated in blocks of five examples, the starting method was counterbalanced across participants, and participants were not told which system generated a question.
- A method × position term shows no detectable order effect (p = 0.41).
b. Consistency of the gains across participants
The table below reports the held-out results (Tab. 20), three tests of the differences, and the number of participants who favour ReferClaw (App. E.13):
| Metric | ReferClaw | Clarify-or-Answer | Difference | Mixed-effects log-odds [95% CI] | McNemar p | Cluster-bootstrap 95% CI | Participants favouring ReferClaw |
|---|---|---|---|---|---|---|---|
| Intent resolved (%) | 80.0 | 70.0 | +10.0 | 0.62 [0.28, 0.96] | 0.012 | [+3.1, +16.9] | 9 / 10 |
| Answer accuracy (%) | 72.0 | 57.3 | +14.7 | 0.71 [0.37, 1.05] | 0.004 | [+6.5, +22.9] | 9 / 10 |
| Response time (s) | 5.1 | 6.9 | −1.8 | – | – | – | 10 / 10 |
| Questions rated confusing (%) | 4.3 | 12.7 | −8.4 | – | – | – | 9 / 10 |
All columns use the 150 held-out examples except the last, which uses all 30 examples per method for each participant.
- The mixed-effects model has crossed random intercepts for participant and example, McNemar tests pair outcomes on the same example, and the cluster bootstrap resamples participants and examples.
- ReferClaw's questions are shorter (20.6 vs. 24.0 tokens) and contrast two grounded candidates, which may contribute to the faster responses and fewer confusing questions.
- Per-participant accuracy differences range from −3.3 to +26.7 points.
Official Comment by Authors
Thank you again for your careful review. We hope our responses address your concerns. Learning still gains 5.3 to 6.1 points when users give brief, forgetful, or free-form corrections, and a free-intent pilot shows a margin over Clarify-or-Answer similar to that of the assigned-intent study. We also clarified that the reporting and promotion probes are disjoint, reported results with sparse feedback, and detailed the counterbalanced assignment and per-participant consistency of the human study. We are grateful for your time and for your recognition of the problem we address and of our controlled analysis.
Response to Reviewer Fc1x (1/3)
We sincerely thank the reviewer for the valuable time spent on our paper and the constructive comments. We appreciate the reviewer's acknowledgement of the significance of our problem setting and controlled analysis. We address the major concerns below and also reflect them in the revision (blue). Table, section, and appendix numbers refer to the PDF.
W1. Fairly ideal conditions
We understand the concern. In short: when users give brief corrections, forget details, or reply in their own words (simulated), learning still gains 5.3–6.1 points, and when participants express their own intentions (small pilot), the margin over Clarify-or-Answer stays similar.
a. Learning without the correct intent and annotated evidence
In the default setting, learning receives the correct intent and annotated evidence after every prediction, which follows MetaClaw's online protocol (Sec. 4.1) and common practice in learning from interaction. We remove them one at a time under the same stream and update rules (new in App. D.3). The table below gives the accuracy of ReferClaw (Full) in each setting (Tab. 8):
Gain: accuracy gain over frozen Base (66.2%), in points. Probe: accuracy on the held-out reporting probe (W2/Q1).
b. Users who give brief corrections, forget details, or reply in their own words
We thank the reviewer for suggesting these settings. We simulate these users on 50% of user utterances:
The first table below applies these noise types to the user's corrections during learning (verdict + one-sentence correction, Tab. 11):
Probe and Gain: as in §a.
The second table applies them to the user's replies to the assistant's clarification questions, with frozen Base and no learning (Tab. 21):
Cost: normalized respondent cost of the questions on an asked input (App. B.2).