acceptodds
Under review as a conference paper at ICLR 2027

When History Helps and Hurts: Selective History Use across Multimodal Turns

Abstract

Reliable multimodal interaction depends on selective use of conversational history: an earlier question may remain relevant while its previous answer is outdated, whereas a current request may depend on historical evidence despite conflicting new observations. Existing multi-turn evaluations rarely separate these history-use demands from underlying question difficulty. To address this gap, we introduce ReTurn, a benchmark of 7,000 base tasks spanning visual and audio evidence for evaluating selective history use. For task-carrying history, Reconfirm/Reground require applying a historical question to current media while varying historical agreement; for evidence-carrying history, Retrieve/Rebind require answering a current question using historical media while varying current-media competition. Each pair preserves the target question, media, and answer. Tasks support open-ended and multiple-choice evaluation, with matched single-turn counterparts serving as answerability references. Across 13 omni-modal, vision-language, and audio-language models, median model-level open-ended accuracy falls from 93.7% with direct input to 72.3% in conversation. Behavioral probes show that high question recall can coexist with weaker task application, while competing media can redirect answers away from historical targets. Supervised adaptation yields only partial gains. ReTurn provides a controlled framework for assessing whether multimodal models select and use the historical information required by each request.

Primary area:
datasets and benchmarks
Keywords:
multi-modal, multi-turn, selective history use

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_cgAFOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces ReTurn, a benchmark designed to evaluate the ability of multimodal models to selectively utilize historical information during multi-turn conversations. The benchmark comprises 7,000 tasks incorporating both visual and audio evidence, and distinguishes between two types of historical dependency: applying historical questions to current media and using historical media to answer current questions. Four task types are constructed using a pairing method that alters either the consistency of historical answers or the level of distractors in the current media, while keeping the target question, target media, and correct answer unchanged.

Critical strengths

The paired construction offers significant evaluative value. By holding the target question, target media, and answer constant while varying either the historical consistency or the current competing media, the study isolates specific variables. This approach minimizes the confounding influence of question difficulty on results and refines multi-turn evaluation into an analysis of two distinct types of historical dependency. This design represents the paper's primary contribution, rendering the observed conflict effects more informative than aggregate multi-turn accuracy alone.

Behavioral experiments reveal failure modes often obscured by average accuracy metrics. Replacing competing media results in minimal changes to target accuracy but causes a marked shift in answers toward the new competing media; additionally, rephrasing options significantly improves performance on certain multiple-choice questions (MCQs). These observations highlight the models' sensitivity to context and the answer interface, providing concrete directions for future evaluation.

Critical weaknesses

Appendix A and Table 5 indicate that, among the 5,000 test tasks, 3,072 involved at least one input medium sharing a parent video with the development data, while 1,875 shared a parent video with inputs used for actual gradient-based training; furthermore, 1,028 test tasks had previously belonged to the TRAIN/DEV partitions. Baseline development relied heavily on Qwen3-Omni’s performance in single-turn and multi-turn scenarios, steering tasks toward higher direct answerability and measurable historical effects. While these facts do not prove that the test tasks were directly used for training, they clearly demonstrate that the final test results are constrained by exposure during the development phase. Re-partitioning based on task IDs or fact groups does not automatically eliminate media exposure or the adaptation of task construction processes to the development data.

The pairings of answers and media sources are primarily validated by the model itself, without human annotation to verify consistency. The media verification described in Appendix D covers only 64 tasks and 31 distinct questions requiring evidence, failing to comprehensively address competing media endpoints. Since a key premise of ReTurn is that different media sources genuinely support the specified consistent or conflicting answers, merely confirming a subset of target answers is insufficient to fully validate this premise. Furthermore, consistent responses across models cannot substitute for independent verification against media sources.Some results are highly sensitive to the scoring process. In Table 11, the OpenQA degradation for MiniCPM-o 4.5 shifts from −0.9 percentage points under rule-based scoring to 32.5 percentage points under content-based scoring. Such a significant change necessitates a specific explanation from the authors regarding which responses were corrected, the reasons for these corrections, and whether the corrections underwent independent verification. Another audit by a judge simultaneously altered both the judge model and the scoring rules, making it impossible to isolate the impact of each factor. Consequently, existing audits fail to address the core issue of scoring dependency.

Table 3 shows that simply repeating the options boosted Qwen3-Omni’s Reground accuracy from 57.8% to 99.2% and MiniCPM-o 4.5’s from 29.7% to 92.2%. This demonstrates that—at least for these diagnostic samples—so-called severe failures can be largely eliminated by simply altering how the options are presented. Option restatement simultaneously provides cues regarding candidate semantics, letter mappings, and task interpretation; the experiments did not distinguish between these specific effects.

Questions for authors

see weakness If the authors can address the issues raised in the "Weakness" section, I will raise my rating.

6 comments
AuthorsOpenReview ↗

Response to Reviewer cgAF(Q1-Q2)

Dear Reviewer cgAF,

Thank you for identifying the target-preserving paired design as the central contribution and for specifying the evidence needed to assess it. We have strengthened that evidence through source-disjoint retraining, human checks of references and competing endpoints, response-level scoring validation, and fixed-rubric judge comparisons. Below, we connect each result to the design choice it validates and to the capability gap that ReTurn measures.

Q1: How do you address prior TRAIN/DEV exposure of TEST media beyond relabeling tasks?

A1: We retain the original TEST identities and prior role/exposure records, and test training-source overlap at the level of actual input parent videos. To directly test the effect of training-source overlap, we completed source-disjoint retraining while retaining the complete TEST and all Long Retrieve/Rebind cases. Filtering by all actual input parent videos leaves 388 development base tasks (302 TRAIN and 86 VAL), with 0.00% actual-input parent overlap across TRAIN, VAL, and TEST. Separation is achieved on the development side without removing evaluation tasks. This experiment isolates the sources used for new training and validation; the benchmark’s historical development exposure remains documented.

Training uses only 6.04% as many base tasks as TEST, yet improves multi-turn performance: Pooling V/A and both interfaces, multi-turn accuracy improves by 1.41 pp for Qwen and 11.01 pp for MiniCPM; single-turn accuracy changes by −0.06 and −0.12 pp, respectively. Both models train from their original bases with a fixed budget; each checkpoint generates its own native histories and is compared with a runtime-matched base.

The table reports multi-turn accuracy gains (SFT−base, pp) by Short/Long and operation, pooling V/A and OpenQA/MCQ using correct counts and denominators. Each cell contains 1,250 interface views; the two interfaces share base tasks and are not independent tasks.

ModelHorizonReconfirmRegroundRetrieveRebind
Qwen3-OmniShort-0.16+0.08-0.32+4.24
Qwen3-OmniLong-1.60+5.04+0.64+3.36
MiniCPM-o 4.5Short+1.68+8.24+1.84+34.64
MiniCPM-o 4.5Long+13.60+10.00+3.84+14.24

The strongest gains occur in the conflict conditions, while single-turn performance stays stable. This pattern connects the adaptation result to ReTurn’s design: the matched tasks expose difficulty in using history, and a small amount of source-disjoint supervision improves that behavior on the complete TEST. Preserving the evaluation set makes the improvement directly comparable across models, operations, and history horizons.

Q2: Does Qwen3-Omni-guided task selection bias the findings or limit generality?

A2: We use Qwen-guided calibration to construct challenge tasks with answerable target facts and diagnostically useful history conditions; the task distribution reflects that objective. We then test whether the difficulty transfers to other model families on the frozen tasks, using each model’s supported modalities and native histories. The exclusion results and newly added proprietary model directly test that cross-model coverage.

The effect is not limited to the construction models. After excluding Qwen3-Omni and Doubao, all 11 remaining models have positive conversational degradation in both interfaces over their supported modalities. The newly added Gemini-3.5-Flash-Lite, evaluated on the full TEST, also drops from 93.96% to 68.16% in OpenQA and from 97.96% to 74.92% in MCQ.

The frozen-TEST exclusion check is shown below, using each model’s supported modalities:

Following the manuscript, conversational degradation is Δconv=Accsingle−Accmulti\Delta_{\mathrm{conv}}=\mathrm{Acc}_{\mathrm{single}}-\mathrm{Acc}_{\mathrm{multi}}, reported in percentage points (pp). Single-turn supplies the complete question and target media directly; multi-turn evaluates the final request after model-generated historical replies.

Exclusion conditionRetained modelsOpenQA Δconv\Delta_{\mathrm{conv}} range (pp)MCQ Δconv\Delta_{\mathrm{conv}} range (pp)
Exclude Qwen3-Omni and Doubao116.72–45.167.00–55.88
Additionally exclude all Qwen-family models815.56–45.1616.85–55.88

These results establish a history-use difficulty across model families, including the newly evaluated proprietary model. Our design makes that comparison informative by fixing the target question, media, and answer within each pair: local answerability is controlled while the surrounding history changes. ReTurn thus provides a focused test of task inheritance and historical-evidence use, with cross-model results showing the reach of this capability gap.

AuthorsOpenReview ↗

Response to Reviewer cgAF(Q3-Q5)

Q3: Can human media review verify answer consistency/conflict at target and competing endpoints?

A3: We extended actual-media verification from target facts to weak competing endpoints, directly checking whether non-target media reveal the target answer and adjudicating flagged cases against the media. Below we report human review by three senior reviewers of reference facts and competing endpoints.

Reference-review measureResult
Human reference checks (three reviewers)Shared target facts underlying the full benchmark
Average agreement on definite shared-subset judgments98.94%
Reference/question/granularity revisions: full benchmark1.37%
Reference/question/granularity revisions: TEST1.60%
Overall Δconv\Delta_{\mathrm{conv}} change under evaluated revisions−0.28 to +0.82 pp
MiniCPM-o OpenQA: before → after32.46 → 32.30 pp

For weak-competition conditions, three-reviewer human media review directly checks whether the non-target endpoint reveals the target answer, followed by media adjudication of flagged cases.

Human competitor reviewResult
Definite judgments without target-answer cues97.22%
Definite judgments flagging possible answer cues2.78%

Reference verification and competitor review check target-answer support and answer cues in non-target media, respectively, providing direct human, media-grounded evidence for the paired design.

Q4: Which responses account for MiniCPM-o’s large rule-to-content scoring change, why were they corrected, and were those corrections independently verified?

A4: We traced MiniCPM-o’s scoring change to valid single-turn answers that literal rules reject. Our content scoring recognizes the answer fact, and independent human review by three senior reviewers validates these corrections. Actual responses below show the rule, content, and human decisions side by side.

QuestionReferenceActual responseRuleContentAll three reviewers
What type of vehicle is visibly depicted?truckA truck.IncorrectCorrectCorrect
What number of minutes is explicitly spoken?2020 minutes.IncorrectCorrectCorrect
How many people are persistently visible in this clip?one1CorrectCorrectCorrect
Which word is spoken first?hiTheIncorrectIncorrectIncorrect

On the complete TEST, the two scoring methods give:

ScoringSingle-turn accuracyMulti-turn accuracyΔconv\Delta_{\mathrm{conv}}
Literal rules6.88%7.76%−0.88 pp
Content scoring93.08%60.62%32.46 pp

All three independent senior reviewers accept all sampled MiniCPM-o responses rejected by rules but accepted by content scoring. Average human–automatic agreement is 93.98% on the definite-judgment subsets.

These results directly support the MiniCPM-o scoring corrections. The broader six-model audit also yields 93.98%–100.00% average human–automatic agreement, providing further support for evaluation reliability.

Q5: Can you separate judge-model and rubric effects in the alternative audit and verify robustness to scoring choices?

A5: Yes. We repeated the comparison on stratified response views using the same production rubric and the same anonymized question, reference, aliases, and response for both judges. We retain the original sampling weights.

Evaluated modelGPT weighted Δconv\Delta_{\mathrm{conv}}Doubao weighted Δconv\Delta_{\mathrm{conv}}
MiniCPM-o 4.532.54 pp39.24 pp
Qwen3-Omni17.68 pp17.99 pp
Qwen3.8-Omni-Flash10.00 pp9.94 pp

Raw verdict agreement between the two automated judges is 90.00%–92.50% across model–condition groups, and all three evaluated models retain positive Δconv\Delta_{\mathrm{conv}}. Because the scoring instructions and inputs are held fixed, this comparison directly isolates the effect of changing the judge. Together with the high human–automatic scoring agreement reported above, these results support the validity of our evaluation and show that the observed multi-turn losses persist across human and automated assessments. The complementary checks strengthen the credibility of the paper’s behavioral conclusions.

AuthorsOpenReview ↗

Response to Reviewer cgAF(Q6)

Q6: What do Reground failures reveal if option restatement fixes them, and which cues drive recovery?

A6: The large recovery identifies a task-inheritance gap: the same locally answerable target becomes difficult when the model must carry the complete option set through history, and restoring that task information improves performance. We test this interpretation with the question/options factorial and separate mapping rotations, measuring recovery of task information and selection under changed letter assignments.

ModelNeither restatedQuestion onlyOptions onlyBoth
Qwen3-Omni57.81%72.66%99.22%98.44%
MiniCPM-o29.69%42.19%92.19%96.09%

The matched Reground results show partial recovery from restoring the question. Even with the complete question supplied, restoring options adds 25.8 pp for Qwen and 53.9 pp for MiniCPM. This provides the full candidate set without identifying the correct candidate. The result demonstrates a task-inheritance requirement beyond retaining the question text alone.

We further change the displayed option mappings on a separate Reconfirm/Reground cohort. Accuracy under the identity mapping and two rotations is 99.22%, 98.83%, and 99.61% for Qwen and 96.48%, 96.09%, and 94.92% for MiniCPM, showing successful semantic selection under the current mapping. Restoring candidates also supplies semantic task cues, so the factorial measures their joint contribution; the remapping experiment adds a direct check of selection under changed letter assignments.

This is the value of separating task inheritance from local answerability: a model can recognize the target answer yet fail to carry the complete task through the conversation. Option restoration makes that gap actionable by identifying information that recovers performance. Thank you for the detailed feedback. High human–automatic scoring agreement and consistent losses under human assessment support the validity of our evaluation. Verified references and endpoints, fixed-rubric judge comparisons, cross-model replication, and gains from limited source-disjoint training provide complementary support for the paired design. Together, these experiments strengthen the paper’s evidence and persuasiveness, showing how ReTurn can expose history-use difficulties and guide interventions that improve performance.

Reviewer_gCAaOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces ReTurn, a benchmark for evaluating selective history use in multimodal conversations. It contains 7,000 base tasks organized into 3,500 matched pairs. The four settings test whether models can reuse an earlier question with new media (Reconfirm/Reground) or answer a question about earlier media (Retrieve/Rebind). Within each pair, the target question, media, and answer are held fixed, while historical agreement or competition from current media varies. The authors evaluate 13 models under both OpenQA and MCQ, using direct-input single-turn runs as references; multi-turn runs keep each model's own historical replies in context. Accuracy generally drops in conversational settings, and the larger drop is often observed under conflicting context. Follow-up experiments look at question recall, option inheritance, and interference from competing media. Inference-time interventions and supervised fine-tuning can help, but only partially.

Critical strengths

1.The paper addresses an important problem: models have to reuse the relevant parts of history without being misled by outdated answers or competing information. The split between task-carrying and evidence-carrying history is, in my view, a helpful way to break the problem down. Because the matched pairs hold the target fixed and only change the surrounding context (Sec. 3, Fig. 2), they allow more focused comparisons than overall multi-turn accuracy alone would. 2.The evaluation covers 13 models, different modalities, short and long conversations, and two answer formats. The authors also go beyond the main results and investigate possible reasons for model failures. The large gains from restating MCQ options (Table 3) suggest that carrying answer choices across turns can be challenging. The media-replacement experiments in Sec. 5 are also telling: models drift toward the competing evidence even when target accuracy barely moves. Results like these go further than the general observation that performance degrades in multi-turn settings.

Critical weaknesses

1.The benchmark leans heavily on automated verification and scoring, and there is no systematic human validation. The media-grounded audit covers only 31 distinct necessary-evidence questions (Appendix D), which feels thin for a benchmark of this size. Table 11 also shows sizable gaps between rule-based and reviewed scoring. For instance, MiniCPM-o 4.5's OpenQA single-to-multi-turn gap changes from -0.9 to 32.5 percentage points between rule-based and reviewed content scoring. The two scoring methods are clearly not equivalent. That said, the alternative-judge audit still shows degradation for the three OpenQA models examined. Even so, a representative human check of the annotations and the reported scores would give me more confidence in the numbers.

2.The task categories are useful, but a few of the analyses do not fully pin down why models fail. The option-restatement gains in Table 3, for example, show that option inheritance matters, but they lump together option recall, letter mapping, and task interpretation; it is hard to tell which one is doing the work. For Retrieve/Rebind, the competing-media experiments show that current media can pull predictions away from the target. What they do not show is how often a correct answer actually requires the historical target media, as opposed to information already present in the earlier text. The authors do take steps to reduce answer copying and provide source-fact probes, but a direct control for historical-media dependence would make this interpretation stronger.

Questions for authors
  1. Could you provide a small but representative human evaluation of the annotations and scoring? Checking media-grounded answers, and especially cases where rule-based and reviewed content scores disagree, would be most informative. Do the main findings hold up under human assessment?
  2. For Retrieve/Rebind, what happens if the historical target media is masked while the historical text is kept? With appropriate controls for input structure and load, this would help quantify how much these tasks genuinely depend on historical multimodal evidence rather than textual cues.
3 comments
AuthorsOpenReview ↗

Response to Reviewer gCAa(Q1-Q3)

Dear Reviewer gCAa,

Thank you for the suggestions. We added historical-media masking, human validation, and option-restoration controls to test task inheritance and historical-evidence use.

Q1: Can you provide a representative human evaluation of the annotations, checking that the answers are supported by the actual media, beyond the limited original audit?

A1: Human media checks by three senior reviewers now cover shared target facts underlying the full benchmark. The table summarizes identified corrections and their impact; the main finding remains stable.

Conversational degradation is Δconv=Accsingle−Accmulti\Delta_{\mathrm{conv}}=\mathrm{Acc}_{\mathrm{single}}-\mathrm{Acc}_{\mathrm{multi}} (pp). Single-turn supplies the complete question and target media; multi-turn uses the model’s native historical replies.

Reference-review measureResult
Human reference checks (three reviewers)Shared target facts underlying the full benchmark
Average agreement on definite shared-subset judgments98.94%
Reference/question/granularity revisions: full benchmark1.37%
Reference/question/granularity revisions: TEST1.60%
Overall Δconv\Delta_{\mathrm{conv}} change under evaluated revisions−0.28 to +0.82 pp
MiniCPM-o OpenQA: before → after32.46 → 32.30 pp

Across the evaluated revisions, overall conversational degradation changes by at most 0.82 pp. This stability supports the robustness of the history-use difficulty under corrected references.

Q2: Can representative human checks validate rule–content scoring disagreements and confirm the main findings?

A2: Three independent reviewers assessed stratified response samples, focusing on rule–content disagreements. The MiniCPM-o examples show valid answers rejected by literal rules.

QuestionReferenceActual responseRuleContentAll three reviewers
What type of vehicle is visibly depicted?truckA truck.IncorrectCorrectCorrect
What number of minutes is explicitly spoken?2020 minutes.IncorrectCorrectCorrect
How many people are persistently visible in this clip?one1CorrectCorrectCorrect
Which word is spoken first?hiTheIncorrectIncorrectIncorrect

On the complete TEST, the two scoring methods give:

ScoringSingle-turn accuracyMulti-turn accuracyΔconv\Delta_{\mathrm{conv}}
Literal rules6.88%7.76%−0.88 pp
Content scoring93.08%60.62%32.46 pp

Across six models, average human–automatic agreement is 93.98%–100.00% on responses with definite judgments from both. For MiniCPM-o, all three reviewers accept all sampled responses rejected by rules but accepted by content scoring.

Human assessment thus supports the recognition of valid answer expressions and reproduces the resulting multi-turn loss. This evidence supports the evaluation’s validity and strengthens the paper’s main finding.

Q3: The option-restatement gains combine option recall, letter mapping, and task interpretation. Can you distinguish which of these factors drives the recovery?

A3: We use a question/options factorial and a separate letter-mapping rotation control. Option restoration supplies both candidates and task cues, so its gain reflects their joint contribution.

ModelNeither restatedQuestion onlyOptions onlyBoth
Qwen3-Omni57.81%72.66%99.22%98.44%
MiniCPM-o29.69%42.19%92.19%96.09%

With the complete question already supplied, restoring options adds 25.8 pp for Qwen and 53.9 pp for MiniCPM on matched Reground tasks, without revealing the correct candidate.

On a separate Reconfirm/Reground cohort, accuracy under the original mapping and two rotations is 99.22%/98.83%/99.61% for Qwen and 96.48%/96.09%/94.92% for MiniCPM. This supports semantic selection under changed letter assignments.

Together, these controls show that preserving the question alone is insufficient: restoring the candidate set supplies additional task information that recovers performance. The rotation results support selection under a changed mapping, but do not separately quantify candidate recall and semantic task cues.

AuthorsOpenReview ↗

Response to Reviewer gCAa(Q4)

Q4: With input structure/load controlled, how does masking historical target media while retaining text affect Retrieve/Rebind?

A4: On matched Retrieve/Rebind tasks covering V/A and Short/Long, we preserve all historical assistant text and mask target media while retaining slots, shapes, and durations. We also test current non-target masks and both masks together.

Each cell reports multi-turn accuracy for original history → historical target masked → current non-target masked, pooling Short/Long.

ModelInterfaceRetrieveRebind
Qwen3-OmniOpenQA85.94% → 14.84% → 89.06%53.12% → 3.91% → 89.06%
Qwen3-OmniMCQ96.88% → 34.38% → 97.66%55.47% → 3.91% → 97.66%
MiniCPM-o 4.5OpenQA84.38% → 19.53% → 91.41%32.03% → 1.56% → 89.84%
MiniCPM-o 4.5MCQ95.31% → 36.72% → 92.97%32.03% → 3.91% → 93.75%

Masking historical targets reduces OpenQA accuracy by 60.16 pp for Qwen and 47.66 pp for MiniCPM despite retaining the text; correct-to-incorrect transitions far exceed the reverse.

Conversely, masking current non-target media improves accuracy. Both controls preserve media slots and dimensions while changing the available evidence or competition. These opposite effects show that historical media contribute beyond the retained text on this cohort, while current competing media can interfere with their use.

Together, human validation and the targeted controls strengthen the evidence for ReTurn’s distinction between task inheritance and historical-evidence use.

Dear Reviewer gCAa,

thank you again for your feedback. Our responses above include additional human validation, option-mapping controls, and historical-media masking experiments. We would appreciate your feedback on whether these address your concerns, or whether any points need further clarification. These additions will also be incorporated into the revised manuscript.

Reviewer_ptsoOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces ReTurn, a benchmark for evaluating selective use of conversational history by multimodal language models. It distinguishes task-carrying history, where an earlier question must be applied to new media, from evidence-carrying history, where a current question must be answered using earlier media.

Critical strengths
  1. The distinction between task-carrying and evidence-carrying history captures two different demands in multimodal interaction.

  2. The benchmark's breadth is valuable for an empirical study of multimodal interaction.

  3. The paper probes question recall, historical-answer neutralization, and media competition. In particular, the Rebind interventions provide evidence that competing media can redirect a model's answer away from the historical target.

Critical weaknesses
  1. Human validation and annotation reliability are insufficiently established. Task construction and verification rely heavily on automated model-based checks. The paper states that human annotation agreement was not measured. The alternative-judge and media-grounded audits are useful sensitivity checks, but they do not substitute for an independently annotated sample with adjudication and inter-annotator agreement. This matters especially for fine-grained visual/audio evidence, answer granularity, and cases where a plausible answer may not be supported by the actual clip.

  2. Reground does not fully isolate historical-media conflict from historical-answer anchoring. The evaluation uses model-generated historical replies, and the paper notes that changing historical media can also change the assistant's earlier replies. Thus, the Reconfirm/Reground contrast can vary both the historical media and the textual answer presented in the conversation. A performance difference may reflect media conflict, anchoring on an earlier model response, or their interaction. The historical-answer neutralization probe is a useful attempt to study this issue, but the reported effects are not sufficiently clean to establish a full separation of these factors.

  3. Short and Long are not universally item-matched. Long task-carrying histories insert additional media-bearing question/answer turns, and evidence-carrying targets may occupy different historical positions. Therefore, performance differences between horizons may reflect additional media, intervening text, target position, or competition. The paper should frame these findings as differences between two conversation configurations rather than a clean causal estimate of history length.

Questions for authors
  1. Can the authors provide operation- and modality-specific results on a strictly source-disjoint subset, and explain how the loss of all Long Retrieve/Rebind cases affects the interpretation of that subset?

  2. Can the authors separate historical-media consistency from historical assistant-answer correctness in a controlled Reground experiment ?

  3. For Reconfirm, how do the authors rule out a shortcut in which the model simply reuses the historical answer, without inspecting the current target media?

2 comments
AuthorsOpenReview ↗

Response to Reviewer ptso(Q1-Q2)

Dear Reviewer ptso,

Thank you for the focused questions about our paired design. ReTurn holds the target task fixed and changes the historical consistency or competing media, creating matched conditions for studying task and evidence use. We have extended this design with crossed media/answer interventions, source-disjoint retraining on the unchanged TEST, and matched-task history-extension analysis. These experiments directly address the distinctions you raised.

Q1: How do source-disjoint results vary by operation/modality, given missing Long Retrieve/Rebind cases?

A1: Thank you for your valuable suggestion. We additionally isolate actual input parent sources on the training side to evaluate source-disjoint training on the complete TEST, including those Long cases.

In this complementary experiment, we use source-disjoint training while retaining the complete TEST and all Long Retrieve/Rebind cases. Filtering by all actual input parent videos leaves 388 development base tasks (302 TRAIN and 86 VAL), with 0.00% actual-input parent overlap across TRAIN, VAL, and TEST. This experiment isolates the sources used for new training and validation; the benchmark’s historical development exposure remains documented.

Training uses only 6.04% as many base tasks as TEST, yet improves multi-turn performance: Qwen gains 1.54/1.28 pp in OpenQA/MCQ, and MiniCPM gains 10.50/11.52 pp. Single-turn accuracy changes by at most 0.32 pp. Both models train from their original bases with a fixed budget; each checkpoint generates its own native histories and is compared with a runtime-matched base.

The table compares the four operations horizontally, reporting multi-turn accuracy gains (SFT−base, pp). Each cell is OpenQA / MCQ; Short/Long are pooled by task count, including every Long task.

ModelModalityReconfirmRegroundRetrieveRebind
Qwen3-OmniV-0.80 / -1.12+4.33 / +5.61+0.80 / +0.48+4.95 / +2.72
Qwen3-OmniA-1.60 / +0.00+1.12 / -0.80-0.96 / +0.32+4.49 / +3.04
MiniCPM-o 4.5V+3.85 / +7.85+5.93 / +3.37+6.07 / +2.72+15.34 / +23.32
MiniCPM-o 4.5A+9.74 / +9.11+11.98 / +15.18+1.60 / +0.96+29.49 / +29.65

The strongest gains occur in the conflict conditions, while single-turn performance stays stable. This pattern connects the adaptation result to ReTurn’s design: the matched tasks expose difficulty in using history, and a small amount of source-disjoint supervision improves that behavior on the complete TEST. Preserving the evaluation set makes the improvement directly comparable across models, operations, and modalities.

Q2: Can you separate historical-media consistency from historical assistant-answer correctness in a controlled Reground experiment?

A2: Yes. We completed a historical-media × first-answer experiment spanning Short/Long and V/A. The current question, current media, target answer, and later conversation are fixed; only the first historical media and assistant text change. “Consistent/conflicting” means that the historical-media answer agrees/disagrees with the current target.

“Neutral acknowledgment” replaces the first assistant answer with exactly “Understood.”: an acknowledgment containing no answer value. The other columns supply the current-target answer value or the conflicting answer value. Historical correctness depends on the media: the current value is correct with consistent media, and the conflicting value with conflicting media. Entries are OpenQA content accuracy.

ModelHorizonHistorical mediaCurrent-answer valueNeutral: Understood.Conflicting-answer value
Qwen3-OmniShortConsistent98.44%96.09%88.28%
Qwen3-OmniShortConflicting97.66%92.19%93.75%
Qwen3-OmniLongConsistent84.77%75.78%77.34%
Qwen3-OmniLongConflicting84.38%71.88%76.17%
MiniCPM-o 4.5ShortConsistent95.31%90.62%87.50%
MiniCPM-o 4.5ShortConflicting93.75%87.50%85.94%
MiniCPM-o 4.5LongConsistent37.89%36.33%37.89%
MiniCPM-o 4.5LongConflicting44.14%37.89%42.97%

We compare media at fixed text and text at fixed media; Long comparisons use the same fixed continuation before averaging within the pair. Current-answer text outperforms neutral acknowledgment in every group, while conflicting text is not uniformly worse than neutral acknowledgment. The factorial therefore distinguishes correct-answer cueing from answer interference instead of collapsing both into one history penalty. This is the diagnostic value of our paired construction: it supplies matched conditions in which the contributions of media and text can be tested directly.

AuthorsOpenReview ↗

Response to Reviewer ptso(Q3-Q5)

Q3: For Reconfirm, how do you rule out the shortcut of simply reusing the historical answer without inspecting the current target media?

A3: We directly test first-answer copying using the consistent-media condition in Q2: historical and current media support the same answer, but the first assistant answer is replaced with an explicitly incorrect conflicting value. All other inputs and later messages stay fixed. Short accuracy remains 88.28% for Qwen and 87.50% for MiniCPM, so these correct responses cannot be explained by unconditional copying.

We use Reconfirm as an answer-consistent control and pair it with Reground, where the same target question requires an updated answer. The copying intervention tests one concrete shortcut; identifying the particular media consulted is a separate question. Together, the paired conditions and the intervention show why Reconfirm is useful: they distinguish successful answer reuse from the ability to update an inherited task when the evidence changes.

Q4: Can independent media annotation validate fine-grained answers, with adjudication and agreement?

A4: We expanded human checks, conducted by three senior reviewers, of whether the actual media support the reference facts and answer granularity, and adjudicated flagged cases against the media. Original judgments and subsequent adjudications are retained separately. The table reports average reference-review agreement and the effect of corrections on the benchmark results.

Following the manuscript, conversational degradation is Δconv=Accsingle−Accmulti\Delta_{\mathrm{conv}}=\mathrm{Acc}_{\mathrm{single}}-\mathrm{Acc}_{\mathrm{multi}}, reported in percentage points (pp). Single-turn supplies the complete question and target media directly; multi-turn evaluates the final request after model-generated historical replies.

Reference-review measureResult
Human reference checks (three reviewers)Shared target facts underlying the full benchmark
Average agreement on definite shared-subset judgments98.94%
Reference/question/granularity revisions: full benchmark1.37%
Reference/question/granularity revisions: TEST1.60%
Overall Δconv\Delta_{\mathrm{conv}} change under evaluated revisions−0.28 to +0.82 pp
MiniCPM-o OpenQA: before → after32.46 → 32.30 pp

On the shared review subset, the reviewers agree on 93 of 94 target facts with definite judgments (average agreement 98.94%); this agreement applies to the jointly reviewed subset, not duplicate review of the entire benchmark. Across evaluated corrections, overall conversational degradation changes by at most 0.82 pp, supporting the stability of the main finding.

Separately, response-level review by the reviewers yields 93.98%–100.00% average human–automatic agreement across six models. Reference and response reviews support data quality and scoring reliability, respectively, strengthening the target-controlled comparison.

Q5: Do Short/Long differences reflect conversation configuration rather than history length alone?

A5: We adopt the distinction you identify: Short and Long are conversation configurations, and we additionally test the contribution of task composition. We match existing TEST inputs on the full question, answer/options, necessary evidence, and first and final user inputs, without selecting on final performance. The table further restricts the comparison to cases with the same first native assistant answer and reports multi-turn accuracy:

ModelInterfaceShortLong
Qwen3-OmniOpenQA91.67%77.87%
Qwen3-OmniMCQ85.37%76.48%
MiniCPM-oOpenQA81.33%45.21%
MiniCPM-oMCQ65.41%48.11%

The identical-first-answer subset covers 100% of Qwen’s matched pool and 76.39% and 93.15% of MiniCPM’s OpenQA/MCQ matched pools. Long histories add media and intervening dialogue together, so the experiment measures history extension as a conversation configuration. Crucially, we match the question, target evidence, and initial answer: the remaining decline demonstrates a history-use challenge even after these task-composition differences are controlled.

Thank you for these questions. High human–automatic scoring agreement supports the reliability of the evaluation. The new controls separate media and answer-text contributions, retain the full evaluation under source-disjoint training, and demonstrate difficulty under matched-task history extension. Together, these experiments strengthen the paired design as a diagnostic and make the paper’s conclusions more convincing.

Reviewer_vx43OpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces ReTurn, a benchmark that evaluates whether multimodal models use conversational history selectively. In the benchmark, the final request depends on history in one of two ways. With task-carrying history, the model must apply an earlier question to the current media. Reconfirm and Reground differ only in whether the historical media supports the same answer as the current media or a different one. With evidence-carrying history, the model must answer a current question using historical media. Retrieve and Rebind differ only in how strongly the current non-target media competes with the historical target.

Each matched pair fixes the target question, target media, answer, and options. A matched single-turn input (question plus target media supplied directly) serves as an answerability reference. The gap between the two, Δconv = Acc_single − Acc_multi, is meant to isolate the cost of using history from the intrinsic difficulty of the question.

The benchmark contains 7,000 base tasks (3,500 pairs), balanced across visual and audio evidence and across Short (2 turns) and Long (5 turns) conversations. Each task has an OpenQA view and an MCQ view. Clips are drawn from FineVideo and LLaVA-Video-178K. Questions and answers are verified against the media by Doubao-Seed-2.0-Lite. Responses are scored with frozen rules, and rule-rejected answers receive a blind review by GPT-5.6 Sol.

The authors evaluate 13 omni-modal, vision-language, and audio-language models. Median OpenQA accuracy drops from 93.7% (single-turn) to 72.3% (multi-turn). The conflict conditions produce the largest drops; for example, median Long OpenQA accuracy on Rebind is 19.9%. Behavioral probes add two findings: models can recall the inherited question yet fail to apply it, and swapping the competing media redirects answers toward the new competitor. Two mitigations are tested. Inference-time restatement/reweighting/routing and LoRA SFT on Qwen3-Omni (+3.42 pp OpenQA, +2.36 pp MCQ) both yield only partial gains.

Critical strengths
  1. Well-motivated problem with a clean decoupling design. The motivation is interesting and close to everyday use: users routinely re-ask an earlier question about new media or ask about something they shared earlier. The paper separates two ways the final request can depend on history (applying a historical question to current media vs. asking a current question about historical media). Each condition is paired with a target-preserving counterpart and a single-turn reference. Together these let the evaluation of history use be decoupled from the difficulty of the underlying question. This matters for my assessment because, as Table 1 also suggests, existing multi-turn benchmarks do not offer this separation. It makes the within-pair contrasts (Reconfirm vs. Reground, Retrieve vs. Rebind) interpretable.

  2. Sufficient novelty and a valuable research question. The framing of history as both necessary information and a source of interference is novel. The results show a large and consistent gap between single-turn answerability and conversational performance across model families. This is a useful finding for the community and supports the value of the benchmark.

Critical weaknesses
  1. The reliability of scoring and verification is not adequately established. This is my main concern, because the headline numbers depend on it.

No human validation. Rule-rejected answers are re-scored by GPT-5.6 Sol, but no human cross-check exists anywhere in the pipeline. Appendix A states that "human annotation agreement was not measured," and Appendix D states that "No human review is included." The LLM review is decisive for some models. Table 11 shows that rule-based and reviewed Δconv can differ drastically. Examples: MiniCPM-o 4.5 OpenQA, −0.9 vs. 32.5 pp; Qwen3.7-Plus MCQ, 42.0 vs. 9.6 pp; MiniCPM-V 4.5 OpenQA, −8.1 vs. 15.6 pp. For these models, whether a conversational drop exists at all is determined by an unvalidated judge. The alternative-judge comparison is unclear. The Doubao-Seed-2.0-Lite audit (Table 12) uses a simplified prompt, so the judge and the rubric change at the same time, and the authors acknowledge this. The disagreements are not small. All 56 are originally-correct answers rejected by the alternative judge. Qwen3-Omni's single-turn accuracy falls from 94.60% to 82.79%, and its Δconv from 17.34 to 10.82 pp. Because the disagreement is one-directional, judge leniency or bias may well be shaping the final conclusions. Verifier/evaluatee overlap. Doubao-Seed-2.0-Lite verifies questions and gold answers, is one of the evaluated models, and also serves as the alternative judge. Qwen3-Omni was the primary model used during benchmark development (Appendix A). Both roles create a risk of bias, especially in cross-model comparisons and rankings.

  1. The templated dialogue structure undercuts the claimed closeness to everyday use. All dependencies are expressed through explicit ordinal references ("the first question", "the first clip I shared"). Long conversations are built by inserting intervening media Q&A turns. This departs from the paper's main selling point of being close to everyday scenarios. Requests such as "For this new clip, answer the first question again" are themselves of questionable meaning: real users would usually rephrase or restate what they want. The routed mitigation policy also depends on parsing these explicit references. Finally, the wording-sensitivity test is insufficient (Table 19). It covers one model, 128 tasks, and two fixed rewrites, and accuracy actually decreases (112 → 106/107). This suggests the results are sensitive to phrasing, but the test is too small to characterize that sensitivity.
Questions for authors
  1. Human validation of the judge. Can the authors provide a human-annotated subset, stratified by model and condition, and report judge–human agreement? This is most important for the models with large rule–content Δconv gaps in Table 11. Can they also estimate the gold-answer error rate with human checks?

  2. Judge comparison under a matched rubric. Can the alternative-judge audit be repeated with the exact production rubric? Reporting agreement among several independent judges would let the reader attribute disagreements to the judge rather than to the prompt.

  3. Verifier bias. Do the conclusions, particularly model rankings and Δconv, change if Doubao-Seed-2.0-Lite and Qwen3-Omni are excluded or reported separately? Do they change if verification uses a different model or a multi-model consensus?

  4. Templated requests. What is the motivation for requests such as "answer the first question again"? Have the authors evaluated more natural reference forms, such as content-based descriptions ("the clip with the dog"), pronouns ("that one"), or paraphrased follow-ups? Can the wording-sensitivity study be extended to more models and samples?

  5. Model coverage. The evaluated set is relatively narrow and lacks recent frontier proprietary models. Would the authors be willing to add more models, e.g., gemini-3.5-flash-lite?

8 comments
AuthorsOpenReview ↗

Response to Reviewer vx43(Q1-Q2)

Dear Reviewer vx43,

Thank you for your careful assessment. We designed ReTurn to compare the same target task under different history conditions, so that local answerability and successful use of history can be evaluated separately. Your questions concern the reliability and reach of that comparison. We address them with human validation, a fixed-rubric judge comparison, broader natural references, and the requested full-TEST Gemini evaluation.

Q1: Can you report stratified human–judge agreement by model and condition, especially for large rule–content gaps?

A1: We added human assessment of responses stratified by model and condition, with three senior reviewers independently checking six models and particular attention to rule–content disagreements. The table reports average agreement on the same responses with definite human and automatic judgments, followed by concrete examples explaining the scoring corrections.

ModelAverage human–automatic agreement
MiMo-V2.597.80%
Qwen3.7-Plus100.00%
Audio Flamingo 3100.00%
MiniCPM-o 4.593.98%
MiniCPM-V 4.594.27%
Qwen3-Omni95.83%

Following the manuscript, conversational degradation is Δconv=Accsingle−Accmulti\Delta_{\mathrm{conv}}=\mathrm{Acc}_{\mathrm{single}}-\mathrm{Acc}_{\mathrm{multi}}, reported in percentage points (pp). Single-turn supplies the complete question and target media directly; multi-turn evaluates the final request after model-generated historical replies.

On the complete TEST, MiniCPM-o scores change as follows, showing how undercounting valid single-turn answers changes measured conversational degradation:

ScoringSingle-turn accuracyMulti-turn accuracyΔconv\Delta_{\mathrm{conv}}
Literal rules6.88%7.76%−0.88 pp
Content scoring93.08%60.62%32.46 pp

Actual MiniCPM-o responses illustrate the scoring distinction. The examples include both correct and incorrect answers: a full phrase or an explicit unit can express the same answer fact.

QuestionReferenceActual responseRuleContentAll three reviewers
What type of vehicle is visibly depicted?truckA truck.IncorrectCorrectCorrect
What number of minutes is explicitly spoken?2020 minutes.IncorrectCorrectCorrect
How many people are persistently visible in this clip?one1CorrectCorrectCorrect
Which word is spoken first?hiTheIncorrectIncorrectIncorrect

For Qwen3.7-Plus MCQ, the change instead comes from deterministic option parsing. With the correct option “A. zero”, the actual response “A. zero” fails the letter-only strict check but is correctly accepted by the option parser. No additional GPT acceptances contribute to its 42.04→9.56 pp change in Δconv\Delta_{\mathrm{conv}}.

Average human–automatic scoring agreement is high across all six evaluated models (93.98%–100.00% on responses with definite judgments), supporting the validity of our evaluation protocol. Together with the response-level examples, this evidence supports the recognition of valid answer expressions and strengthens confidence in the reported scores.

Q2: Can you estimate the gold-answer error rate through human checks?

A2: We expanded human media checks, conducted by three independent senior reviewers, to cover the shared target facts underlying the full benchmark. The table reports the proportion of tasks affected by reference-answer, question, or answer-granularity corrections identified through those checks: 1.37% of all tasks and 1.60% of TEST. Across the evaluated revisions, overall Δconv\Delta_{\mathrm{conv}} changes by at most 0.82 pp, preserving the main finding.

Reference-review measureResult
Human reference checks (three reviewers)Shared target facts underlying the full benchmark
Average agreement on definite shared-subset judgments98.94%
Reference/question/granularity revisions: full benchmark1.37%
Reference/question/granularity revisions: TEST1.60%
Overall Δconv\Delta_{\mathrm{conv}} change under evaluated revisions−0.28 to +0.82 pp
MiniCPM-o OpenQA: before → after32.46 → 32.30 pp

We have therefore connected benchmark-wide reference verification to its measured effect on the results. The stability of conversational degradation after correction strengthens our central finding: the history conditions reveal a difficulty even when the target fact is held fixed.

AuthorsOpenReview ↗

Response to Reviewer vx43(Q3-Q4)

Q3: Can you report independent-judge agreement under the exact production rubric to separate judge and prompt effects?

A3: Yes. We repeated the comparison on stratified response views using the same production rubric and the same anonymized question, reference, aliases, and response for both judges. We retain the original sampling weights.

Evaluated modelGPT weighted Δconv\Delta_{\mathrm{conv}}Doubao weighted Δconv\Delta_{\mathrm{conv}}
MiniCPM-o 4.532.54 pp39.24 pp
Qwen3-Omni17.68 pp17.99 pp
Qwen3.8-Omni-Flash10.00 pp9.94 pp

Raw verdict agreement between the two automated judges is 90.00%–92.50% across model–condition groups, and all three evaluated models retain positive Δconv\Delta_{\mathrm{conv}}. Because the scoring instructions and inputs are held fixed, this comparison directly isolates the effect of changing the judge. Together with the high human–automatic scoring agreement reported above, these results support the validity of our evaluation and show that the observed multi-turn losses persist across human and automated assessments. The complementary checks strengthen the credibility of the paper’s behavioral conclusions.

Q4: Are rankings and degradation robust to excluding construction models or changing verifiers?

A4: We address model overlap and the verification procedure separately. For model overlap, the frozen-TEST exclusion check below examines the retained models’ rankings and Δconv\Delta_{\mathrm{conv}}; every retained model has a positive Δconv\Delta_{\mathrm{conv}} in both interfaces. For verification reliability, we added independent human media checks. We have not rebuilt the benchmark with another verifier or a multi-model consensus.

Exclusion conditionRetained modelsOpenQA Δconv\Delta_{\mathrm{conv}} range (pp)MCQ Δconv\Delta_{\mathrm{conv}} range (pp)
Exclude Qwen3-Omni and Doubao116.72–45.167.00–55.88
Additionally exclude all Qwen-family models815.56–45.1616.85–55.88

Even after excluding all Qwen-family models and Doubao, OpenQA Δconv\Delta_{\mathrm{conv}} remains 15.56–45.16 pp and MCQ Δconv\Delta_{\mathrm{conv}} 16.85–55.88 pp. This is precisely why we center ReTurn on within-model, target-matched comparisons: each model answers the same target question with the same target evidence, while the history condition changes. The resulting losses recur across model families, with human verification and matched-rubric scoring providing independent support. The exclusion check preserves retained models’ scores and relative rankings on the frozen benchmark; its key contribution is to establish the broad model coverage of the finding.

AuthorsOpenReview ↗

Response to Reviewer vx43(Q5)

Q5: Why repeat the first question? Can broader wording tests cover natural references and follow-ups?

A5: We use requests such as “answer the earlier question again” to fix the inherited task, making history use comparable at the same target. Following your suggestion, we expanded OpenQA evaluation to two models and nine expression families across all four operations and V/A, preserving task media, native histories, and scoring. These now include actual visual/audio content descriptions alongside pronoun-style follow-ups, earlier-question quotations, and discussion-topic references.

The nine expression templates are listed together below. Q5 and Q6 use exactly the same 576 expression views, single-turn baselines, and unmodified multi-turn results per model, covering 512 distinct base tasks.

N01  Minimal ellipsis        Now?
N02  Current-clip follow-up  What would your answer be for this one?
N03  Earlier-task reference  Let’s use my opening question for this clip.
N04  Question-topic cue      About the animal’s fur colour in my opening
                             question—what about this one?
N05  Unique earlier clip     Let’s return to the earlier clip. [Q]
N06  Time/position cue       Go back to the second clip I sent. [Q]
N07  Earlier-question cue    Take the clip from my earlier question,
                             “[earlier Q]”. [Q]
N08  Discussion-topic cue    The clip from our discussion about the
                             speaker’s first name—[Q]
N09  Media-content cue       For this clip, answer the question I asked about
                             the earlier clip showing a studio microphone
                             before black acoustic foam panels.
                             In the earlier clip showing a man seated on
                             leather sofa beside an acoustic guitar: [Q]

Each cell gives multi-turn content accuracy (%), with the drop from the matched single-turn baseline in parentheses (pp); each operation has 144 views. Overall single-turn accuracy is 96.88% for Qwen and 93.40% for MiniCPM, identical to Q6.

ModelOverallReconfirmRegroundRetrieveRebind
Qwen3-Omni67.36 (↓29.51)77.78 (↓20.83)65.97 (↓32.64)90.97 (↓4.17)34.72 (↓60.42)
MiniCPM-o 4.564.58 (↓28.82)63.19 (↓31.25)77.78 (↓16.67)84.03 (↓8.33)33.33 (↓59.03)

[Q] denotes the complete current question and [earlier Q] an actual earlier question. Rewrites of one task remain related views. N01/N02/N05 use Short histories; the other families cover Short/Long. Minimal ellipsis and near-reference require a unique referent; N07/N08 use their unambiguous anchor subsets.

For N09, 64 tasks were selected before inference, balanced across V/A, Short/Long, and the four operations. Each description was checked against the actual historical media for support, a unique referent, and absence of answer leakage. The content cues identify either the inherited question or the historical target. The table uses the unmodified results from the matched Q6 runs; the full wording sweep of 3,040 views per model is retained as supplementary evidence and is not pooled into this table.

All four operations retain a single-to-multi-turn loss, with the largest drops in Rebind and smaller drops in Retrieve. Across nine expression families, the results extend the finding beyond ordinal wording and show that the difficulty varies by history-use operation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.