acceptodds
Withdrawn from ICLR 2027

What Should We Ask Next? Retrieval-Aware Question Learning for Interactive ReID

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Interactive retrieval under partial evidence is a sequential information-acquisition problem: an agent must decide which question will create the most useful evidence for the next retrieval update. Existing systems train this decision by imitating an offline ordering of candidate QA pairs, although question value is determined by the response it elicits and its downstream effect on retrieval. We establish that candidate discriminativeness and perceived usefulness provide weak supervision for this objective, then introduce RAVEL, a retrieval-aware online reinforcement learning framework for interactive person re-identification. RAVEL initializes from supervised question generation, observes the current Top-4 candidates directly, and optimizes the question policy with rank feedback from the full question–answer–retrieval loop. Experiments on Interactive-PEDES show that RAVEL delivers progressively stronger retrieval performance across five interaction rounds. Further analysis shows that RAVEL reallocates the questioning budget toward localized open-ended attributes, which provide more useful retrieval evidence and yield the largest gains on initially difficult queries.

Primary area:
reinforcement learning
Keywords:
Interactive Person Re-Identification, Retrieval-Aware Question Learning, Multimodal Question Generation, Online Reinforcement Learning, Sequential Information Acquisition, Vision-Language Models, Text-Based Person Re-Identification

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_v71oOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies Interactive ReID, where a questioner must select questions based on the evolving candidate set and dialogue history. It proposes RAVEL, a retrieval-aware online reinforcement learning framework that initializes from supervised question generation, directly observes the current Top-4 candidates, and optimizes the question policy using rank feedback from the complete question–answer–retrieval loop. Experiments on Interactive-PEDES and three additional text-based ReID benchmarks report improved retrieval performance, along with analyses of the learned questioning behavior.

Critical strengths
  1. The paper frames interactive ReID as a partially observable sequential process, where the value of a question depends on the evidence produced by its answer and the resulting retrieval update. Evaluating question utility through realized retrieval gain rather than static judgments provides a coherent formulation of the problem.

  2. The empirical study reveals a weak association between static question judgments and realized retrieval gain, highlighting the limitations of using static supervision to assess question utility. It also provides useful diagnostics of how the learned policy changes its questioning behavior across interaction rounds.

  3. The experiments fix the retriever and answerer during policy learning and evaluation, which helps isolate the effect of the questioning strategy within a controlled interaction environment. The additional experiments with alternative retrieval backbones provide evidence that the learned policy is not entirely tied to a single retriever.

Critical weaknesses
  1. The Qwen3-VL analysis finds only a weak association between static question judgments and realized retrieval gain. However, this does not show that offline question ordering is ineffective or that online policy learning is necessary. The motivation for retrieval-aware online reinforcement learning is therefore insufficient. 2. Table 3 lacks a direct comparison between SFT with the validity gate and SFT without RL. Since removing the gate causes invalid-action collapse, it remains unclear whether the improvement comes from retrieval-aware RL or from the gate constraining the policy to answerable questions.

  2. Table 3 lacks a direct comparison between SFT with the validity gate and SFT without RL. Since removing the gate causes invalid-action collapse, it remains unclear whether the improvement comes from retrieval-aware RL or from the gate constraining the policy to answerable questions.

Questions for authors
  1. Are new states generated during policy updates dynamically added to the curated 3,000-state training pool, and if not, how does the current method fully justify the claim of true online learning?

  2. Could the authors provide a sensitivity analysis for the manually fixed reward coefficients in Equation 9 to clarify how much the final performance relies on these specific values?

1 comment

Round awareness and objective. RAVEL rewards the immediate retrieval change for each question-answer step; it does not optimize a return summed across all rounds. Each policy input explicitly includes the round index, current Top-4, and dialogue. Training samples states from different rounds, allowing the LLM to recognize the current round and tailor its question to the stage and evidence. This is our claim of round-aware question generation under a per-step objective. The shared explanation appears in the Overall Response under “Sequential policy and reward horizon.”

Validity gate and Table 3. We corrected the submitted statement that the gate was applied during both training and test-time rollouts. Section 3.4 now says it is used only during RL training, and Table 3's Gate column indicates validity filtering; Supplementary B lists whole-outfit and multi-region rules. Thus, the SFT-only condition and all evaluations have no gate. The Overall Response under “Invalid-question reward is −0.2” reports the audit: broad questions fell from 3,955/36,865 turns (10.73%) in the no-gate transcript to 70/36,865 (0.19%) in the gate-trained ungated evaluation. Removing the remaining answers changed R@1 from 81.581 to 81.500.

Online learning and the fixed state pool. During training, each update samples a question, obtains an answer and rank feedback from the retriever, and updates the questioner. These action outcomes come from fresh interactions during learning; offline RL instead learns from logged actions and outcomes without new interaction. Whether the starting-state pool expands is a separate design choice (Wagenmaker & Pacchiano, 2023). The 0K/1K/3K/5K scores and Supplementary updates appear in the Overall Response under “State-pool scaling”; Supplementary B describes the pool and its construction.

Updated reward and scaling results. The revised reward removes the submitted 0.35 Rank-1 entry term. From the shared no-interaction R@1 of 37.61, the five-round result is 81.76/94.83/97.52/48.18 (R@1/R@5/R@10/mAP; BRI 0.484). Under this reward, transfer results are 82.57/95.96/98.02/73.38 on CUHK-PEDES, 71.30/86.23/91.07/43.63 on ICFG-PEDES, and 73.88/91.64/95.86/57.80 on RSTPReid. Retriever transfer to RDE reaches 86.19/96.85/98.56/49.82 (BRI 0.416). The revised reward was used for the main experiment, dataset and retriever transfer, and training ablations. The Overall Response under “Updated reward and main result” and “Controlled ablations” reports the full results and manuscript changes.

The state-pool comparison reports 0K at 69.44/87.78/92.80/45.48/0.698, 1K at 79.71/93.88/96.96/48.85/0.536, 3K at 81.76/94.83/97.52/48.18/0.484, and 5K at 81.161/94.982/97.423/48.328/0.489 (R@1/R@5/R@10/mAP/BRI). We updated the supplementary state-pool table and construction text to include all four pool sizes. The full comparison appears under “State-pool scaling” in the Overall Response.

Invalid-question reward (−0.2). The no-gate transcript contained 3,955 broad whole-outfit or multi-region prompts in 36,865 turns (10.73%). Removing their answers lowered R@1 from 84.42 to 75.38. After adding these rules to the RL training gate, a gate-trained policy produced only 70 broad whole-outfit questions (0.19%) in its ungated evaluation. Removing those answers changed R@1 from 81.581 to 81.500. This comparison supports the intended role of the penalty: discouraging broad prompts that bypass the localized-question protocol. The Overall Response under “Invalid-question reward is −0.2” gives the transcript audit and full gate-run results; Section 3.4 now states that the gate is used only in RL training, and Supplementary B lists the invalid-question rules.

Within-group repeat penalty (−0.02). GRPO compares eight sampled questions for one state. If all eight are duplicates with the same retrieval outcome, their group-relative advantages are all zero. The first occurrence retains its retrieval reward, while later duplicates receive −0.02, giving tied samples a small relative signal and encouraging question diversity. In the main 3K run, groups averaged 7.93 unique questions out of 8, with a 9.20% repetition rate. Since the deduction applies only to duplicates and is small relative to retrieval changes, a fine-grained coefficient sweep would be difficult to interpret. The full rationale is in the Overall Response under “Within-group repeat penalty is −0.02.”

Reviewer_bu3KOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes RAVEL, a retrieval-aware online reinforcement learning framework for interactive person re-identification (IReID). The analysis first shows that previous work (LLaVA-ReID) uses imitation learning to learn a retrieval-aware question generator, which is not optimal, as discriminativeness and perceived usefulness correlate weakly with actual rank improvement. The paper then proposes an online RL framework (RAVEL) to directly learn a policy that maximises retrieval performance. Experiments show clear gains with RL compared to the baselines.

Critical strengths
  1. The motivation of this paper is clear: LLaVA-ReID is a retrieval-aware question generator trained using greedy, one-step supervision. The use of imitation learning from greedy question labels doesn't directly optimise a long-term sequential decision policy. Further, this paper’s analysis shows that a question that looks useful is not necessarily a question that improves retrieval. Motivated by this, the paper’s RAVEL uses an online reinforcement learning framework that trains the Questioner using the actual retrieval performance obtained after asking questions and receiving answers
  2. The ablation studies are generally sound, which clearly shows the design choices: RL brings +5 Rank-1 gain over LLaVA-ReID, and the validity gate prevents collapse
Critical weaknesses
  1. While RAVEL introduces an online RL framework for IReID, it’s essentially an immediate, per-question retrieval improvement. RAVEL does not establish that the learned policy optimises long-horizon rewards across the complete multi-round trajectories.
  2. This paper lacks an analysis to justify the reward coefficient choice (0.35)
  3. Minor: I partially disagree with the description at L38-L41 regarding LLaVA-ReID. Based on my understanding, LLaVA-ReID indeed decomposes fine-grained descriptions into candidate question–answer pairs and uses a greedy look-forward strategy to order them by immediate ranking improvement. However, LLaVA-ReID can adapt its questions during inference, and it does not simply generate questions based on a fixed list. However, I agree that LLaVA-ReID relies on offline training rather than directly optimising the outcomes of the interaction process.
Questions for authors
  1. The experimental settings and benchmarks largely follow LLaVA-ReID. However, why are the baseline numbers reported lower than in the original LLaVA-ReID paper?
  2. Why use 0.35 as the reward coefficient? Is this number empirically determined?
2 comments

Sequential objective. RAVEL's reward is one-step and greedy: it optimizes the immediate rank change for each sampled question, not a sum over the conversation. The policy input explicitly includes the round index, current Top-4, and dialogue, and training samples states from different rounds. The LLM can recognize the current round and condition its question on that stage and the evidence available. This is our claim of round-aware sequential question learning; the objective does not optimize cumulative return over a full dialogue. The shared explanation of this scope appears in “Sequential policy and reward horizon” in the Overall Response.

Reward coefficient and updated results. The revised reward removes the submitted 0.35 Rank-1 entry term. From the shared no-interaction R@1 of 37.61, the five-round result is 81.76/94.83/97.52/48.18 (R@1/R@5/R@10/mAP; BRI 0.484). Under this reward, transfer results are 82.57/95.96/98.02/73.38 on CUHK-PEDES, 71.30/86.23/91.07/43.63 on ICFG-PEDES, and 73.88/91.64/95.86/57.80 on RSTPReid. Retriever transfer to RDE reaches 86.19/96.85/98.56/49.82 (BRI 0.416). The revised reward was used for the main experiment, dataset and retriever transfer, and training ablations. The Overall Response under “Updated reward and main result” and “Controlled ablations” reports the full results and manuscript changes.

The revised behavior analysis finds that 96.5% of first-round questions ask about the surroundings and 89.6% of their answers contain scene cues. Later questions shift toward localized appearance and action details, while Rank-1 rises from 37.61 to 59.50 after Round 1 and 81.76 after Round 5. The consolidated discussion of RAVEL's five-round question profile and retrieval trajectory is also presented under “Behavior and retrieval progression” in the Overall Response. LLaVA-ReID's first-round profile is different: posture/action appears in 46.0% of questions, shoes in 15.5%, and environment in 7.8% under the same lexical tags. We updated Section 4.2 and Figure 5 to present the five-round topic and retrieval progression and the matched LLaVA-ReID first-round profile.

LLaVA-ReID inference description. At inference, LLaVA-ReID generates context-aware questions conditioned on the dialogue and selected candidate-person images. Its learned visual selector chooses representative candidate images to supply as context. The one-step looking-forward procedure provides supervision for question generation during training. The broader motivation for distinguishing annotation-derived supervision from retrieval-feedback policy learning is discussed under “Motivation and prior work” in the Overall Response. We revised the Introduction to describe LLaVA-ReID's inference-time question generation and visual selector. The training description remains in Related Work.

LLaVA-ReID baseline. The original LLaVA-ReID paper reports 73.20 Rank-1 at Round 5 with a 40-token answer limit. We reran the baseline with that cap and obtained around 63–65 Rank-1, consistent with the 63.827 Round-5 result in GitHub Issue #10.

At 40 tokens, some answers are truncated before they state a useful attribute, weakening the retrieval text. Raising the cap to 64 increases our LLaVA-ReID result to 67.94. We use 64 as the maximum answer budget because answers accumulate over five rounds, and substantially longer outputs can exceed IRRA's 192-token input limit. Every method, including RAVEL, uses the same 64-token cap.

Supplementary B now states this shared cap explicitly. Table 1 reports the LLaVA-ReID result under this protocol.

Reviewer_bu3KOpenReview ↗

Response to authors

Thanks authors for providing additional explanation and for the revision of the paper. This has largely addressed my questions. I'm willing to keep the score as 3 - Weak acceptance.

Reviewer_vGbzOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work proposes a retrieval-aware question learning framework, RAVEL, under the interactive person re-identification problem. Unlike previous methods relying on offline question ordering, RAVEL uses supervised fine-tuning followed by GRPO to optimize question generation based on retrieval rank improvements after question-answer interactions. Experiments on Interactive-PEDES demonstrate improved retrieval performance over existing baselines, with additional analyses of question quality, candidate conditioning, and transferability across datasets and retrievers.

Critical strengths
  • The paper addresses a well-motivated problem in interactive retrieval by highlighting the limitations of offline question ordering and incorporating retrieval feedback into question learning.

  • The proposed framework demonstrates consistent retrieval improvements across multiple interaction rounds, with additional transfer experiments on different datasets and retrieval backbones.

  • The paper provides analyses of question quality, candidate conditioning, and question types, along with ablation studies supporting the effectiveness of the proposed framework.

Critical weaknesses
  • The methodological novelty is somewhat limited, as the proposed framework primarily combines existing SFT, GRPO, and retrieval-based reward optimization. The technical contributions beyond applying these established techniques to interactive person re-identification problem are not sufficiently clarified. The unique challenges identifed by the authors and technical novelty needs to be more highlighted.

  • The paper indeed emphasizes sequential question learning, but the training objective primarily optimizes immediate retrieval improvements from sampled interaction states. Thus, it remains unclear whether the learned policy captures long-term question planning beyond greedy per-round optimization, which raises concerns about the mismatch between conceptual novelty and technical contribution.

  • The evaluation relies on a fixed LLM-based simulated answerer, without sufficient analysis of robustness to different answerers or noisy and incomplete user responses. This raises concerns about the practical applicability of the proposed framework in real-world interactive retrieval. Human validation and qualitative analysis can strengthen the contributions and practicality of this work.

Questions for authors

Please see weaknesses

1 comment

Novelty and contribution. The key distinction is the training signal and question space: LLaVA-ReID’s one-step look-ahead scores prepared QA pairs and uses a high-gain pair as supervision, whereas RAVEL generates free-form questions and learns from the retrieval change observed after each answer. Reviewer bu3K finds the motivation clear, while Reviewer vGbz asks whether the weak association in our diagnostic establishes the need for online learning. These observations address different aspects: LLaVA-ReID’s results show that its supervision is useful; our diagnostic asks whether static judgments predict realized retrieval gains.

Given the current Top-4 and dialogue, RAVEL samples a question. The answerer responds, the retriever updates the ranking, and the observed rank change supplies the reward for a GRPO policy update (Shao et al., 2024). The policy thus learns from outcomes of its own free-form questions, including questions beyond the prepared annotations. This task-specific coupling of question generation and retrieval feedback is our central contribution to IReID.

The broader rationale for this distinction and the role of the diagnostic are discussed in “Motivation and prior work” in the Overall Response. In the revision, the Introduction describes LLaVA-ReID’s inference-time question generation and visual selector; the account of offline question supervision remains in Related Work.

Question rounds and reward. RAVEL's GRPO objective is one-step and greedy: each sampled question-answer step receives an immediate rank-change reward, not a return summed over the dialogue. Every policy input includes the round index, current Top-4, and dialogue. Since training samples states from different rounds, the LLM can recognize the current round and tailor its question to that stage and the evidence collected so far. This is our sequential-learning claim: round-aware question generation under a per-question objective. The shared explanation of this scope appears in “Sequential policy and reward horizon” in the Overall Response.

Reward coefficient and updated results. Removing the 0.35 Rank-1 entry term gives 81.76/94.83/97.52/48.18 at Round 5 (R@1/R@5/R@10/mAP; BRI 0.484), from the shared no-interaction R@1 of 37.61. We applied the revised reward to the main experiment, dataset transfer, retriever transfer, training-stage ablations, and completed state-pool comparisons. The revised equation and full transfer and scaling results are summarized in the Overall Response under “Updated reward and main result,” “Controlled ablations,” and “State-pool scaling.”

Fixed answerer. The two reviews highlight complementary aspects of the same choice. Reviewer v71o values a fixed answerer and retriever because they isolate the effect of the question policy; Reviewer vGbz asks whether the result generalizes to other answerers or human and noisy responses. The same fixed setup therefore answers the first question through a controlled question-policy comparison, while leaving response robustness as a distinct question. The shared protocol and rationale are detailed in the Overall Response under “Controlled answerer.”

Across questioner comparisons, we hold the Qwen2.5-7B-Instruct answerer, answer budget, and retriever constant. The answerer extracts a fact from a short witness description in response to a short question, a task current instruction-tuned models are expected to solve with near-ceiling accuracy. We expect tests with noisy, incomplete, or human responses to be more informative than simply replacing one answerer model with another. Rebuttal time went to the additional dataset, retriever, ablation, and available scaling experiments; we plan to study other answerers and human responses when time permits. Supplementary B now explicitly states that every method uses a 64-token answer cap.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Overall Response

We thank all three reviewers for their thoughtful feedback. We appreciate Reviewer bu3K's recognition of the motivation and results, Reviewer vGbz's emphasis on retrieval feedback for interactive question learning, and Reviewer v71o's observation that fixing the answerer and retriever isolates the question policy. Together, these perspectives clarify both the value of RAVEL's interaction-based learning and the scope of the current evaluation.

Motivation and prior work. LLaVA-ReID already generates context-aware questions at inference from the dialogue and selected candidate images, and its results show that its offline supervision is useful. Our distinction is the training signal. LLaVA-ReID's one-step look-ahead uses prepared QA annotations: GPT-4o-generated fine-grained descriptions are divided into short sub-captions, each paired with a QA example. This supervision is useful, but its coverage is bounded by the details selected in those descriptions; a question about one captioned detail need not be the most useful question for the current Top-4. To guide our learning objective, we tested whether static judgments of discriminativeness or perceived usefulness predict retrieval gain. They correlate only weakly with observed rank changes. This finding led us to train RAVEL from the retrieval change observed after each answer. Its free-form questioner can explore questions beyond those represented in the prepared annotations.

Sequential policy and reward horizon. RAVEL's current GRPO objective is one-step and greedy: each sampled question-answer step receives an immediate retrieval-gain reward. Every policy input includes the round index, current Top-4, and dialogue. Because training samples states from different rounds, the LLM can recognize the current interaction stage and tailor its generated question to that round and the evidence available so far. Our sequential-learning claim is round-aware question generation, with retrieval feedback at each sampled step.

Updated reward and main result. The revised reward removes the 0.35 Rank-1 entry term. From the 37.61 no-interaction R@1, RAVEL reaches 74.08/91.21/95.21/43.59 at Round 3 and 81.76/94.83/97.52/48.18 at Round 5 (R@1/R@5/R@10/mAP), with BRI 0.484. We reran the main, transfer, training-ablation, and retriever-transfer experiments under this reward. In the manuscript, we revised the reward equation and updated the main, transfer, and ablation tables with these results.

Across CUHK-PEDES, ICFG-PEDES, and RSTPReid, RAVEL+IRRA obtains 82.57/95.96/98.02/73.38, 71.30/86.23/91.07/43.63, and 73.88/91.64/95.86/57.80, respectively (R@1/R@5/R@10/mAP). Replacing IRRA with RDE after training yields 86.19/96.85/98.56/49.82 and BRI 0.416. We updated the transfer table and the retriever-ablation table with these results.

Controlled ablations. For candidate input (R@1/R@5/R@10/mAP/BRI), LLaVA-ReID with its selector scores 67.94/86.82/92.47/45.16/0.703, direct Top-4 input scores 67.84/87.54/92.72/45.33/0.701, and RAVEL with direct Top-4 scores 81.76/94.83/97.52/48.18/0.484. The training ablation reports 44.12/68.24/77.73/32.55/1.075 for the native questioner, 69.44/87.78/92.80/45.48/0.698 for SFT only, 67.42/86.79/92.93/46.04/0.696 for RL-only with the gate, and 81.76/94.83/97.52/48.18/0.484 for SFT followed by gated RL; the ungated SFT-to-RL run collapses. We updated Tables 3 and 4 with these training, candidate-input, and retriever comparisons. The text accompanying Table 3 now clarifies that Gate indicates validity filtering, while Section 3.4 states that the gate is used only during RL training.

State-pool scaling. The completed state-pool comparison reports (R@1/R@5/R@10/mAP/BRI): 0K, 69.44/87.78/92.80/45.48/0.698; 1K, 79.71/93.88/96.96/48.85/0.536; 3K, 81.76/94.83/97.52/48.18/0.484; and 5K, 81.161/94.982/97.423/48.328/0.489. We added the completed 5K result to the supplementary state-pool table and updated its construction description to include the 1K and 5K comparison pools.

AuthorsOpenReview ↗

Overall Response(2)

Behavior and retrieval progression. Across 7,373 test queries, 96.5% of RAVEL's first-round questions ask about the surroundings and 89.6% of first-round answers contain scene cues. Rank-1 rises from 37.61 before interaction to 59.50 after Round 1 and 81.76 after Round 5. Under the same lexical tags, LLaVA-ReID's first-round questions focus more on posture/action (46.0%) and shoes (15.5%), with 7.8% tagged as environment. Across later rounds, RAVEL shifts toward localized appearance and action details. We revised Section 4.2 and Figure 5's discussion and caption to report this five-round topic progression, retrieval trajectory, and matched first-round comparison.

Invalid-question reward is −0.2. Gate training reduced broad whole-outfit or multi-region questions from 10.73% to 0.19%; removing the remaining answers changed R@1 by only 0.081 points.

In the no-gate five-round transcript, 3,955 of 36,865 questions (10.73%) asked for broad whole-outfit or multi-region information. Such a question can elicit several clothing attributes at once and provide unusually broad evidence in one turn. Using the same retriever, removing these answers lowered R@1 from 84.42 to 75.38. This audit identifies the protocol shortcut addressed by the −0.2 invalid-question penalty.

We added explicit whole-outfit and multi-region rules to the RL training gate; these rules flag all 3,955 questions in the no-gate transcript. In a separate gate-trained run, evaluation with the gate off produced 81.581/95.144/97.654/48.235 (R@1/R@5/R@10/mAP; BRI 0.487). Its Round 1–5 R@1 scores were 59.243, 67.910, 74.325, 78.503, and 81.581. The test transcript contained 70 broad whole-outfit questions (0.19% of 36,865 turns); removing their answers changed R@1 from 81.581 to 81.500. The results show that the questioner learned the localized-question protocol while evaluation still tested the questioner itself.

In the revision, Section 3.4 now says the gate is used only during RL training. Supplementary B lists whole-outfit and multi-region requests among the invalid-question rules, and the text accompanying Table 3 clarifies that its Gate column indicates validity filtering.

Within-group repeat penalty is −0.02. The deduction differentiates repeated samples in an otherwise tied GRPO group, addressing the zero-advantage case while affecting only duplicate questions.

GRPO compares eight questions sampled for the same state. If all eight are identical and produce the same retrieval result, their group-relative advantages are all zero. The first occurrence keeps its retrieval reward; only subsequent duplicates receive −0.02, giving otherwise tied samples a small relative signal. In the main 3K run, groups contained 7.93 unique questions out of 8 on average, with a 9.20% repetition rate. Because the deduction applies only to duplicates and is small relative to observed retrieval changes, a fine-grained coefficient sweep would be difficult to interpret; we did not scan it independently.

Controlled answerer. Reviewer v71o views the fixed answerer and retriever as strengths because they isolate the question policy. Reviewer vGbz raises the complementary question of robustness to other answerers and human or noisy responses. The same setup therefore supports a controlled comparison of question policies, while motivating a separate test of response robustness. Across questioner comparisons, we use the same Qwen2.5-7B-Instruct answerer, answer budget, and retriever protocol.

The answerer extracts a fact from a short witness description in response to a short question. We expect current instruction-tuned models to answer this constrained task with near-ceiling accuracy, so tests with noisy, incomplete, or human responses would be more informative than simply replacing one answerer model with another. Rebuttal time went to the additional dataset, retriever, ablation, and available scaling experiments. We plan to study other answerers and human responses when time permits. Supplementary B now explicitly states that every method uses a 64-token answer cap.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.