- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates the impact of image role and option binding in multimodal memory evaluation on model performance. The authors find that the absence of explicit correspondences among the reference image, query image, and candidate images may adversely affect final scores. To address this, the paper proposes the RPS method for the fixed-architecture VisualMem interface, which enhances input organization by incorporating explicit role and binding information, while preserving both the retrieval results and the image sequence. Experiments were conducted across 58 image-option queries, examining various configurations of option ordering, placement of binding information, role phrasing, and generation budgets. The results demonstrate that the efficacy of binding repair varies across models, and the generation budget also influences outcomes. The authors further conducted manual answer extraction and verification, comparative evaluations of retrieval methods across 696 queries, and external image injection experiments to analyze how input organization strategies affect both evaluation scores and the interpretability of retrieval performance.
Critical strengths
- The authors identified specific and practically impactful issues in evaluation interfaces. While keeping the retrieval results and image sequences unchanged, they demonstrated through binding repair experiments, erroneous binding control experiments, and option-position balancing experiments that input serialization can significantly affect the scores of multiple models. This provides valuable diagnostic insights for multimodal memory evaluation.
- The experimental analysis was conducted with due care, thoroughly considering alternative explanations. The paper distinguishes between the effects of binding methods, generation budgets, and answer extraction, and examines whether the processor retains any intervening effects. By transparently reporting inter-model differences, negative findings, and statistical limitations, the authors enhance the credibility of their conclusions within the scope of their study.
Critical weaknesses
- The generalizability of the conclusions remains limited. The core experiments focused on a single interface and 58 baseline tasks. Adding models, budgets, and option permutations did not expand the coverage of independent tasks. Statistical evidence from external experiments and retrieval comparisons is also relatively limited; thus, it remains insufficient to determine the extent of the impact of these issues in broader memory evaluation settings.
- A complete reasoning and scoring pipeline remains difficult to replicate independently. The supplementary materials support the reproduction of key statistical metrics, but still lack certain reasoning dependencies and raw generated outputs. Furthermore, manual answer extraction and verification only covered historical Qwen outputs, excluding subsequent new experiments, which limits the possibility of independent verification of the full evaluation pipeline.
Questions for authors
- Can you provide a minimal working input construction, processor invocation traces, and scoring reproduction paths—including accessible versions of missing dependencies, reconstruction methods for frozen issues, and publicly available new raw output samples? You are not required to redistribute restricted images, but you should enable reviewers to verify the end-to-end process from legally obtained upstream data to predicted labels.
- For recent experiments where many setups still encounter budget exhaustion or unresolved outputs, particularly with external InternVL, do you have additional sufficient output budgets or paired results under controlled answer-output protocols? This would help assess how much of the external RPS benefit can be preserved after improving answer extractability. The focus here is on the interpretability scope of external evidence, rather than demanding uniform conclusions regarding internal mechanisms.
3 comments
Response to Reviewer nVgh——-2
2.A complete reasoning and scoring pipeline remains difficult to replicate independently. The supplementary materials support the reproduction of key statistical metrics, but still lack certain reasoning dependencies and raw generated outputs. Furthermore, manual answer extraction and verification only covered historical Qwen outputs, excluding subsequent new experiments, which limits the possibility of independent verification of the full evaluation pipeline.
2.1 How can reviewers reproduce input construction, inference, and scoring?
The minimal reproducibility package (ICLR2027_Minimal_Reproducibility_11-17.zip) includes pinned dependencies and installable Flash Linear Attention wheels. It also provides scripts to reconstruct frozen questions and build inputs, processor invocation traces, raw responses and generated token IDs from the revised experiments, separate answer keys, and scoring scripts.
The minimal working example uses five upstream images for the VisualMem question persona_8/event_151/q_1/rotation_0. The README explains how to obtain these images under the dataset's terms, verify their hashes, and reconstruct the Native and RPS inputs. It then shows how to inspect the processor calls, run inference using the recorded model revision, and score the extracted answer labels. Dataset images and model weights are obtained from upstream sources.
To verify scoring on CPU, run python reproduce.py verify and python reproduce.py replay --output work/replayed_scores.json. These commands check 208 stored predictions, reported separately as 8 technical examples and 200 sampled outputs from the main experiments. In our pinned-dependency check, the processed inputs and generated token sequences matched the recorded references for the 8 fixed examples. The README documents checkpoint limitations for Gemma and Molmo.
See Section 3.3 and the Reproducibility Statement in the main text, and Supplementary Sections R.1–R.3 and Table S66.
2.2 How did we verify answer extraction on the new experiments?
We added independent human verification of 200 outputs generated with a 1,024-token limit. These outputs come from 143 base questions and cover four models, three question sets, and four types of experiments. Two annotators independently read the original responses and recorded their answers without access to the answer keys or parser outputs. A third annotator reviewed all 200 records.
The two annotators agreed on all 200 answer labels and on 199 of the 200 conflict flags. The parser matched the reviewed human labels on 199/200 outputs (99.5%, descriptive Wilson 95% CI 97.22%–99.91%, Cohen's kappa = 0.9935). The only discrepancy was an explicit answer D that the parser missed. Scoring with human and parser extraction yielded 80 and 79 correct outputs, respectively. Both agreed on the 7 outputs that reached the token limit without giving a final choice.
These results support automated answer extraction for scoring this stratified sample. The human check also distinguishes responses with no final answer from responses whose answers the parser missed.
See Section 4.4 and Table 4 of the main text, and Supplementary Section R.4 and Tables S67–S68.
Response to Reviewer nVgh——-3-4
3.Can you provide a minimal working input construction, processor invocation traces, and scoring reproduction paths—including accessible versions of missing dependencies, reconstruction methods for frozen issues, and publicly available new raw output samples? You are not required to redistribute restricted images, but you should enable reviewers to verify the end-to-end process from legally obtained upstream data to predicted labels.
The supplementary materials include ICLR2027_Minimal_Reproducibility_11-17.zip, our minimal reproducibility package. The same ZIP is available in the public Hugging Face repository. It includes pinned dependencies, installable fla_core and flash_linear_attention wheels (both version 0.6.0), scripts to reconstruct frozen questions and build inputs, processor invocation traces, and raw responses and generated token IDs from the revised experiments.
The README walks through one fixed VisualMem question (persona_8/event_151/q_1/rotation_0). Reviewers obtain its five images from the upstream dataset under its terms and verify their hashes. They can then rebuild the Native and RPS inputs, run the processor and inspect its traces, generate responses with the recorded model revision, and extract and score the predicted labels.
To replay scoring from saved outputs on CPU, run python reproduce.py verify and python reproduce.py replay --output work/replayed_scores.json. These commands check 208 stored predictions, reported separately as 8 technical examples and 200 sampled outputs from the revised experiments.
See Section 3.3 and the Reproducibility Statement in the main text, and Supplementary Sections R.1–R.3 and Table S66.
4.For recent experiments where many setups still encounter budget exhaustion or unresolved outputs, particularly with external InternVL, do you have additional sufficient output budgets or paired results under controlled answer-output protocols? This would help assess how much of the external RPS benefit can be preserved after improving answer extractability. The focus here is on the interpretability scope of external evidence, rather than demanding uniform conclusions regarding internal mechanisms.
We ran two types of paired diagnostics. Raising the output-token limit increased the number of extractable answers in some settings. Under controlled decoding, all answers were extractable in both groups. RPS still scored higher in both tests, although the evidence is limited.
On the original set of 120 questions constructed from MuirBench, we compared Flat and RPS using the same InternVL3.5-8B-HF model, images, and image order. We raised the limit from 512 to 1,024 and 2,048 tokens. The number of extractable answers increased in both groups. Even at 2,048 tokens, only 46/120 answers were extractable under Flat and 54/120 under RPS. RPS scored higher at all three limits. At 2,048 tokens, the gap was 28.33 percentage points. The Holm-adjusted p-value was 0.1875 for each of the three differences. In the new VisioMath diagnostic with InternVL3.5-14B-HF, the number of extractable answers in the adjacent-neutral condition also rose, from 95/300 at 1,024 tokens to 178/300 at 4,096 tokens.
To compare scores when all answers were extractable, we prefilled FINAL= under controlled decoding. The model then selected a valid option letter, and generation ended. On MuirBench, all 120 outputs in each group had extractable answers. RPS scored 25.83 percentage points higher, a descriptive difference on this constructed question set. In the same diagnostic on VisioMath, all answers were extractable under both Native and RPS. RPS scored 7.33 percentage points higher. The difference was not significant after Holm correction (adjusted p = 0.1912).
The score gaps observed when all answers were extractable show that differences in answer extractability cannot fully explain the results under these controlled conditions. Controlled decoding changes the generation process, so we limit the interpretation of these gaps to the diagnostic settings.
See Sections 4.4 and 4.6 and Table 6 of the main text, and Supplementary Sections P.2–P.3 and Tables S60–S62.
Response to Reviewer nVgh——-1
1. The generalizability of the conclusions remains limited. The core experiments focused on a single interface and 58 baseline tasks. Adding models, budgets, and option permutations did not expand the coverage of independent tasks. Statistical evidence from external experiments and retrieval comparisons is also relatively limited; thus, it remains insufficient to determine the extent of the impact of these issues in broader memory evaluation settings.
1.1 New questions and input formats
The main binding experiment in the original manuscript used 58 VisualMem questions with image answer choices. The four cyclic choice orders were repeated tests of the same questions. The revised manuscript retains these 58 questions and adds fixed subsets of 300 questions each from MMR-Life and VisioMath. Both new question sets retain their original choice orders. The main binding experiment now covers 658 base questions across three public answering pipelines.
The question sets also cover different binding formats. The original VisualMem input lacks explicit labels for the image choices. MMR-Life already places choice letters after the candidate images. VisioMath already provides a global mapping. The two new experiments therefore test whether making image roles and choice mappings more explicit helps when some mapping information is already available.
See Section 4.1 and Table 1 of the main text, and Supplementary Section N.1.
1.2 Larger models and a higher output-token limit
We replaced the original Qwen3.5-9B and InternVL3.5-8B-HF with Qwen3.5-27B and InternVL3.5-14B-HF, respectively. We retained Gemma3-4B and Molmo2-8B. For the main binding comparisons, we raised the common output-token limit from 128 to 1,024. We did so to reduce the risk of a response being cut off before an extractable answer appears. Each comparison holds image content and order fixed. The VisualMem comparisons also hold the retrieved evidence fixed.
In this setting, 5 of the 12 comparisons between the original input (Native) and role-preserving serialization (RPS) show gains after Holm correction. Gains pass correction on both newly added question sets. For example, InternVL3.5-14B-HF improves by 28.3 percentage points on MMR-Life, and Qwen3.5-27B improves by 8.3 percentage points on VisioMath. We observe no improvement for Molmo2-8B on any of the three question sets. The gains vary by model and question set.
See Sections 4.1–4.2 and Tables 1–2 of the main text.
1.3 External experiments and retrieval comparisons
The newly added question sets show that explicit binding can affect scores on some multi-image choice tasks. Shared images and task composition still limit generalization. Real long-term interactions require further testing. On 120 questions constructed from MuirBench, controlled decoding made all answers extractable under Flat and RPS. RPS accuracy was still 25.83 percentage points higher. This is a descriptive result for the constructed question set (Section 4.6 and Table 6 of the main text).
The retrieval comparisons use only the 58 VisualMem questions in their original choice order. We hold the evidence fixed within each retrieval configuration. For Gemma3-4B, the accuracy gap between Full and Naive shifts by 20.7 percentage points when we change the serialization. The Holm-adjusted p-value is 0.25. None of the 4 primary or 8 secondary interaction comparisons passes correction. Comparisons of retrieval configurations should therefore report the serialization used. The current evidence is insufficient to determine whether using RPS changes which retrieval configuration performs best.
See Sections 4.5–4.6 and 5, Tables 5–6, and Figure 4 of the main text. In the supplementary material, see Section O.3, Table S59, and Sections Q.1–Q.3.