- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces ReTurn, a benchmark designed to evaluate the ability of multimodal models to selectively utilize historical information during multi-turn conversations. The benchmark comprises 7,000 tasks incorporating both visual and audio evidence, and distinguishes between two types of historical dependency: applying historical questions to current media and using historical media to answer current questions. Four task types are constructed using a pairing method that alters either the consistency of historical answers or the level of distractors in the current media, while keeping the target question, target media, and correct answer unchanged.
Critical strengths
The paired construction offers significant evaluative value. By holding the target question, target media, and answer constant while varying either the historical consistency or the current competing media, the study isolates specific variables. This approach minimizes the confounding influence of question difficulty on results and refines multi-turn evaluation into an analysis of two distinct types of historical dependency. This design represents the paper's primary contribution, rendering the observed conflict effects more informative than aggregate multi-turn accuracy alone.
Behavioral experiments reveal failure modes often obscured by average accuracy metrics. Replacing competing media results in minimal changes to target accuracy but causes a marked shift in answers toward the new competing media; additionally, rephrasing options significantly improves performance on certain multiple-choice questions (MCQs). These observations highlight the models' sensitivity to context and the answer interface, providing concrete directions for future evaluation.
Critical weaknesses
Appendix A and Table 5 indicate that, among the 5,000 test tasks, 3,072 involved at least one input medium sharing a parent video with the development data, while 1,875 shared a parent video with inputs used for actual gradient-based training; furthermore, 1,028 test tasks had previously belonged to the TRAIN/DEV partitions. Baseline development relied heavily on Qwen3-Omni’s performance in single-turn and multi-turn scenarios, steering tasks toward higher direct answerability and measurable historical effects. While these facts do not prove that the test tasks were directly used for training, they clearly demonstrate that the final test results are constrained by exposure during the development phase. Re-partitioning based on task IDs or fact groups does not automatically eliminate media exposure or the adaptation of task construction processes to the development data.
The pairings of answers and media sources are primarily validated by the model itself, without human annotation to verify consistency. The media verification described in Appendix D covers only 64 tasks and 31 distinct questions requiring evidence, failing to comprehensively address competing media endpoints. Since a key premise of ReTurn is that different media sources genuinely support the specified consistent or conflicting answers, merely confirming a subset of target answers is insufficient to fully validate this premise. Furthermore, consistent responses across models cannot substitute for independent verification against media sources.Some results are highly sensitive to the scoring process. In Table 11, the OpenQA degradation for MiniCPM-o 4.5 shifts from −0.9 percentage points under rule-based scoring to 32.5 percentage points under content-based scoring. Such a significant change necessitates a specific explanation from the authors regarding which responses were corrected, the reasons for these corrections, and whether the corrections underwent independent verification. Another audit by a judge simultaneously altered both the judge model and the scoring rules, making it impossible to isolate the impact of each factor. Consequently, existing audits fail to address the core issue of scoring dependency.
Table 3 shows that simply repeating the options boosted Qwen3-Omni’s Reground accuracy from 57.8% to 99.2% and MiniCPM-o 4.5’s from 29.7% to 92.2%. This demonstrates that—at least for these diagnostic samples—so-called severe failures can be largely eliminated by simply altering how the options are presented. Option restatement simultaneously provides cues regarding candidate semantics, letter mappings, and task interpretation; the experiments did not distinguish between these specific effects.
Questions for authors
see weakness If the authors can address the issues raised in the "Weakness" section, I will raise my rating.
6 comments
Response to Reviewer cgAF(Q3-Q5)
Q3: Can human media review verify answer consistency/conflict at target and competing endpoints?
A3: We extended actual-media verification from target facts to weak competing endpoints, directly checking whether non-target media reveal the target answer and adjudicating flagged cases against the media. Below we report human review by three senior reviewers of reference facts and competing endpoints.
| Reference-review measure | Result |
|---|---|
| Human reference checks (three reviewers) | Shared target facts underlying the full benchmark |
| Average agreement on definite shared-subset judgments | 98.94% |
| Reference/question/granularity revisions: full benchmark | 1.37% |
| Reference/question/granularity revisions: TEST | 1.60% |
| Overall change under evaluated revisions | −0.28 to +0.82 pp |
| MiniCPM-o OpenQA: before → after | 32.46 → 32.30 pp |
For weak-competition conditions, three-reviewer human media review directly checks whether the non-target endpoint reveals the target answer, followed by media adjudication of flagged cases.
| Human competitor review | Result |
|---|---|
| Definite judgments without target-answer cues | 97.22% |
| Definite judgments flagging possible answer cues | 2.78% |
Reference verification and competitor review check target-answer support and answer cues in non-target media, respectively, providing direct human, media-grounded evidence for the paired design.
Q4: Which responses account for MiniCPM-o’s large rule-to-content scoring change, why were they corrected, and were those corrections independently verified?
A4: We traced MiniCPM-o’s scoring change to valid single-turn answers that literal rules reject. Our content scoring recognizes the answer fact, and independent human review by three senior reviewers validates these corrections. Actual responses below show the rule, content, and human decisions side by side.
| Question | Reference | Actual response | Rule | Content | All three reviewers |
|---|---|---|---|---|---|
| What type of vehicle is visibly depicted? | truck | A truck. | Incorrect | Correct | Correct |
| What number of minutes is explicitly spoken? | 20 | 20 minutes. | Incorrect | Correct | Correct |
| How many people are persistently visible in this clip? | one | 1 | Correct | Correct | Correct |
| Which word is spoken first? | hi | The | Incorrect | Incorrect | Incorrect |
On the complete TEST, the two scoring methods give:
| Scoring | Single-turn accuracy | Multi-turn accuracy | |
|---|---|---|---|
| Literal rules | 6.88% | 7.76% | −0.88 pp |
| Content scoring | 93.08% | 60.62% | 32.46 pp |
All three independent senior reviewers accept all sampled MiniCPM-o responses rejected by rules but accepted by content scoring. Average human–automatic agreement is 93.98% on the definite-judgment subsets.
These results directly support the MiniCPM-o scoring corrections. The broader six-model audit also yields 93.98%–100.00% average human–automatic agreement, providing further support for evaluation reliability.
Q5: Can you separate judge-model and rubric effects in the alternative audit and verify robustness to scoring choices?
A5: Yes. We repeated the comparison on stratified response views using the same production rubric and the same anonymized question, reference, aliases, and response for both judges. We retain the original sampling weights.
| Evaluated model | GPT weighted | Doubao weighted |
|---|---|---|
| MiniCPM-o 4.5 | 32.54 pp | 39.24 pp |
| Qwen3-Omni | 17.68 pp | 17.99 pp |
| Qwen3.8-Omni-Flash | 10.00 pp | 9.94 pp |
Raw verdict agreement between the two automated judges is 90.00%–92.50% across model–condition groups, and all three evaluated models retain positive . Because the scoring instructions and inputs are held fixed, this comparison directly isolates the effect of changing the judge. Together with the high human–automatic scoring agreement reported above, these results support the validity of our evaluation and show that the observed multi-turn losses persist across human and automated assessments. The complementary checks strengthen the credibility of the paper’s behavioral conclusions.
Response to Reviewer cgAF(Q6)
Q6: What do Reground failures reveal if option restatement fixes them, and which cues drive recovery?
A6: The large recovery identifies a task-inheritance gap: the same locally answerable target becomes difficult when the model must carry the complete option set through history, and restoring that task information improves performance. We test this interpretation with the question/options factorial and separate mapping rotations, measuring recovery of task information and selection under changed letter assignments.
| Model | Neither restated | Question only | Options only | Both |
|---|---|---|---|---|
| Qwen3-Omni | 57.81% | 72.66% | 99.22% | 98.44% |
| MiniCPM-o | 29.69% | 42.19% | 92.19% | 96.09% |
The matched Reground results show partial recovery from restoring the question. Even with the complete question supplied, restoring options adds 25.8 pp for Qwen and 53.9 pp for MiniCPM. This provides the full candidate set without identifying the correct candidate. The result demonstrates a task-inheritance requirement beyond retaining the question text alone.
We further change the displayed option mappings on a separate Reconfirm/Reground cohort. Accuracy under the identity mapping and two rotations is 99.22%, 98.83%, and 99.61% for Qwen and 96.48%, 96.09%, and 94.92% for MiniCPM, showing successful semantic selection under the current mapping. Restoring candidates also supplies semantic task cues, so the factorial measures their joint contribution; the remapping experiment adds a direct check of selection under changed letter assignments.
This is the value of separating task inheritance from local answerability: a model can recognize the target answer yet fail to carry the complete task through the conversation. Option restoration makes that gap actionable by identifying information that recovers performance. Thank you for the detailed feedback. High human–automatic scoring agreement and consistent losses under human assessment support the validity of our evaluation. Verified references and endpoints, fixed-rubric judge comparisons, cross-model replication, and gains from limited source-disjoint training provide complementary support for the paired design. Together, these experiments strengthen the paper’s evidence and persuasiveness, showing how ReTurn can expose history-use difficulties and guide interventions that improve performance.
Response to Reviewer cgAF(Q1-Q2)
Dear Reviewer cgAF,
Thank you for identifying the target-preserving paired design as the central contribution and for specifying the evidence needed to assess it. We have strengthened that evidence through source-disjoint retraining, human checks of references and competing endpoints, response-level scoring validation, and fixed-rubric judge comparisons. Below, we connect each result to the design choice it validates and to the capability gap that ReTurn measures.
Q1: How do you address prior TRAIN/DEV exposure of TEST media beyond relabeling tasks?
A1: We retain the original TEST identities and prior role/exposure records, and test training-source overlap at the level of actual input parent videos. To directly test the effect of training-source overlap, we completed source-disjoint retraining while retaining the complete TEST and all Long Retrieve/Rebind cases. Filtering by all actual input parent videos leaves 388 development base tasks (302 TRAIN and 86 VAL), with 0.00% actual-input parent overlap across TRAIN, VAL, and TEST. Separation is achieved on the development side without removing evaluation tasks. This experiment isolates the sources used for new training and validation; the benchmark’s historical development exposure remains documented.
Training uses only 6.04% as many base tasks as TEST, yet improves multi-turn performance: Pooling V/A and both interfaces, multi-turn accuracy improves by 1.41 pp for Qwen and 11.01 pp for MiniCPM; single-turn accuracy changes by −0.06 and −0.12 pp, respectively. Both models train from their original bases with a fixed budget; each checkpoint generates its own native histories and is compared with a runtime-matched base.
The table reports multi-turn accuracy gains (SFT−base, pp) by Short/Long and operation, pooling V/A and OpenQA/MCQ using correct counts and denominators. Each cell contains 1,250 interface views; the two interfaces share base tasks and are not independent tasks.
The strongest gains occur in the conflict conditions, while single-turn performance stays stable. This pattern connects the adaptation result to ReTurn’s design: the matched tasks expose difficulty in using history, and a small amount of source-disjoint supervision improves that behavior on the complete TEST. Preserving the evaluation set makes the improvement directly comparable across models, operations, and history horizons.
Q2: Does Qwen3-Omni-guided task selection bias the findings or limit generality?
A2: We use Qwen-guided calibration to construct challenge tasks with answerable target facts and diagnostically useful history conditions; the task distribution reflects that objective. We then test whether the difficulty transfers to other model families on the frozen tasks, using each model’s supported modalities and native histories. The exclusion results and newly added proprietary model directly test that cross-model coverage.
The effect is not limited to the construction models. After excluding Qwen3-Omni and Doubao, all 11 remaining models have positive conversational degradation in both interfaces over their supported modalities. The newly added Gemini-3.5-Flash-Lite, evaluated on the full TEST, also drops from 93.96% to 68.16% in OpenQA and from 97.96% to 74.92% in MCQ.
The frozen-TEST exclusion check is shown below, using each model’s supported modalities:
Following the manuscript, conversational degradation is Δconv=Accsingle−Accmulti, reported in percentage points (pp). Single-turn supplies the complete question and target media directly; multi-turn evaluates the final request after model-generated historical replies.
These results establish a history-use difficulty across model families, including the newly evaluated proprietary model. Our design makes that comparison informative by fixing the target question, media, and answer within each pair: local answerability is controlled while the surrounding history changes. ReTurn thus provides a focused test of task inheritance and historical-evidence use, with cross-model results showing the reach of this capability gap.