acceptodds
Under review as a conference paper at ICLR 2027

Auditing Role Collisions in Multimodal Memory Evaluation

Abstract

Multimodal memory evaluation assesses how retrieved evidence supports question answering. However, scores also depend on how this evidence is presented and how responses are generated and scored. We audit a public VisualMem answering path and identify missing explicit role and option bindings, a condition we term a role collision. We introduce role-preserving serialization (RPS) to express these bindings while keeping retrieval outputs fixed. Across four model stacks and all 58 eligible image-choice questions with balanced answer positions, RPS improves accuracy by up to 34.9 percentage points over source-derived serialization, with no gain on Molmo2. To examine how correct bindings affect scores, we separate mapping placement from role wording in a paired 2 × 2 factorial design. On Qwen3.5-9B, raising the output-token limit from 128 to 512 reduces the adjacent-over-global advantages from 25.86 to 5.60 points under neutral wording and from 19.83 to 8.62 under semantic wording; only the neutral reduction passes the expanded Holm correction. By contrast, Gemma and InternVL retain the same answer labels across 64/128/256/512-token runs, while semantic wording lowers InternVL’s global-placement accuracy by 13.79 points. For scoring validation, independent human extraction agrees with the frozen parser on 193/200 archived Qwen outputs. Finally, serialization changes retrieval-method contrasts on 696 questions, while Retrieval-only remains best overall. On 120 MuirBench reference-injection questions, RPS exceeds flat injection by 6.67–27.50 points; these shifts and gains do not pass expanded Holm corrections. Together, these findings show why bindings, output-token limits, and answer-extraction rules must be specified before attributing score differences to retrieval quality.

Primary area:
foundation or frontier models, including LLMs
Keywords:
multimodal memory, vision-language models, evaluation methodology, image-option binding, input serialization, generation budget, answer extraction

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_nVghOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the impact of image role and option binding in multimodal memory evaluation on model performance. The authors find that the absence of explicit correspondences among the reference image, query image, and candidate images may adversely affect final scores. To address this, the paper proposes the RPS method for the fixed-architecture VisualMem interface, which enhances input organization by incorporating explicit role and binding information, while preserving both the retrieval results and the image sequence. Experiments were conducted across 58 image-option queries, examining various configurations of option ordering, placement of binding information, role phrasing, and generation budgets. The results demonstrate that the efficacy of binding repair varies across models, and the generation budget also influences outcomes. The authors further conducted manual answer extraction and verification, comparative evaluations of retrieval methods across 696 queries, and external image injection experiments to analyze how input organization strategies affect both evaluation scores and the interpretability of retrieval performance.

Critical strengths
  1. The authors identified specific and practically impactful issues in evaluation interfaces. While keeping the retrieval results and image sequences unchanged, they demonstrated through binding repair experiments, erroneous binding control experiments, and option-position balancing experiments that input serialization can significantly affect the scores of multiple models. This provides valuable diagnostic insights for multimodal memory evaluation.
  2. The experimental analysis was conducted with due care, thoroughly considering alternative explanations. The paper distinguishes between the effects of binding methods, generation budgets, and answer extraction, and examines whether the processor retains any intervening effects. By transparently reporting inter-model differences, negative findings, and statistical limitations, the authors enhance the credibility of their conclusions within the scope of their study.
Critical weaknesses
  1. The generalizability of the conclusions remains limited. The core experiments focused on a single interface and 58 baseline tasks. Adding models, budgets, and option permutations did not expand the coverage of independent tasks. Statistical evidence from external experiments and retrieval comparisons is also relatively limited; thus, it remains insufficient to determine the extent of the impact of these issues in broader memory evaluation settings.
  2. A complete reasoning and scoring pipeline remains difficult to replicate independently. The supplementary materials support the reproduction of key statistical metrics, but still lack certain reasoning dependencies and raw generated outputs. Furthermore, manual answer extraction and verification only covered historical Qwen outputs, excluding subsequent new experiments, which limits the possibility of independent verification of the full evaluation pipeline.
Questions for authors
  1. Can you provide a minimal working input construction, processor invocation traces, and scoring reproduction paths—including accessible versions of missing dependencies, reconstruction methods for frozen issues, and publicly available new raw output samples? You are not required to redistribute restricted images, but you should enable reviewers to verify the end-to-end process from legally obtained upstream data to predicted labels.
  2. For recent experiments where many setups still encounter budget exhaustion or unresolved outputs, particularly with external InternVL, do you have additional sufficient output budgets or paired results under controlled answer-output protocols? This would help assess how much of the external RPS benefit can be preserved after improving answer extractability. The focus here is on the interpretability scope of external evidence, rather than demanding uniform conclusions regarding internal mechanisms.
3 comments
AuthorsOpenReview ↗

Response to Reviewer nVgh——-1

1. The generalizability of the conclusions remains limited. The core experiments focused on a single interface and 58 baseline tasks. Adding models, budgets, and option permutations did not expand the coverage of independent tasks. Statistical evidence from external experiments and retrieval comparisons is also relatively limited; thus, it remains insufficient to determine the extent of the impact of these issues in broader memory evaluation settings.

1.1 New questions and input formats

The main binding experiment in the original manuscript used 58 VisualMem questions with image answer choices. The four cyclic choice orders were repeated tests of the same questions. The revised manuscript retains these 58 questions and adds fixed subsets of 300 questions each from MMR-Life and VisioMath. Both new question sets retain their original choice orders. The main binding experiment now covers 658 base questions across three public answering pipelines.

The question sets also cover different binding formats. The original VisualMem input lacks explicit labels for the image choices. MMR-Life already places choice letters after the candidate images. VisioMath already provides a global mapping. The two new experiments therefore test whether making image roles and choice mappings more explicit helps when some mapping information is already available.

See Section 4.1 and Table 1 of the main text, and Supplementary Section N.1.

1.2 Larger models and a higher output-token limit

We replaced the original Qwen3.5-9B and InternVL3.5-8B-HF with Qwen3.5-27B and InternVL3.5-14B-HF, respectively. We retained Gemma3-4B and Molmo2-8B. For the main binding comparisons, we raised the common output-token limit from 128 to 1,024. We did so to reduce the risk of a response being cut off before an extractable answer appears. Each comparison holds image content and order fixed. The VisualMem comparisons also hold the retrieved evidence fixed.

In this setting, 5 of the 12 comparisons between the original input (Native) and role-preserving serialization (RPS) show gains after Holm correction. Gains pass correction on both newly added question sets. For example, InternVL3.5-14B-HF improves by 28.3 percentage points on MMR-Life, and Qwen3.5-27B improves by 8.3 percentage points on VisioMath. We observe no improvement for Molmo2-8B on any of the three question sets. The gains vary by model and question set.

See Sections 4.1–4.2 and Tables 1–2 of the main text.

1.3 External experiments and retrieval comparisons

The newly added question sets show that explicit binding can affect scores on some multi-image choice tasks. Shared images and task composition still limit generalization. Real long-term interactions require further testing. On 120 questions constructed from MuirBench, controlled decoding made all answers extractable under Flat and RPS. RPS accuracy was still 25.83 percentage points higher. This is a descriptive result for the constructed question set (Section 4.6 and Table 6 of the main text).

The retrieval comparisons use only the 58 VisualMem questions in their original choice order. We hold the evidence fixed within each retrieval configuration. For Gemma3-4B, the accuracy gap between Full and Naive shifts by 20.7 percentage points when we change the serialization. The Holm-adjusted p-value is 0.25. None of the 4 primary or 8 secondary interaction comparisons passes correction. Comparisons of retrieval configurations should therefore report the serialization used. The current evidence is insufficient to determine whether using RPS changes which retrieval configuration performs best.

See Sections 4.5–4.6 and 5, Tables 5–6, and Figure 4 of the main text. In the supplementary material, see Section O.3, Table S59, and Sections Q.1–Q.3.

AuthorsOpenReview ↗

Response to Reviewer nVgh——-2

2.A complete reasoning and scoring pipeline remains difficult to replicate independently. The supplementary materials support the reproduction of key statistical metrics, but still lack certain reasoning dependencies and raw generated outputs. Furthermore, manual answer extraction and verification only covered historical Qwen outputs, excluding subsequent new experiments, which limits the possibility of independent verification of the full evaluation pipeline.

2.1 How can reviewers reproduce input construction, inference, and scoring?

The minimal reproducibility package (ICLR2027_Minimal_Reproducibility_11-17.zip) includes pinned dependencies and installable Flash Linear Attention wheels. It also provides scripts to reconstruct frozen questions and build inputs, processor invocation traces, raw responses and generated token IDs from the revised experiments, separate answer keys, and scoring scripts.

The minimal working example uses five upstream images for the VisualMem question persona_8/event_151/q_1/rotation_0. The README explains how to obtain these images under the dataset's terms, verify their hashes, and reconstruct the Native and RPS inputs. It then shows how to inspect the processor calls, run inference using the recorded model revision, and score the extracted answer labels. Dataset images and model weights are obtained from upstream sources.

To verify scoring on CPU, run python reproduce.py verify and python reproduce.py replay --output work/replayed_scores.json. These commands check 208 stored predictions, reported separately as 8 technical examples and 200 sampled outputs from the main experiments. In our pinned-dependency check, the processed inputs and generated token sequences matched the recorded references for the 8 fixed examples. The README documents checkpoint limitations for Gemma and Molmo.

See Section 3.3 and the Reproducibility Statement in the main text, and Supplementary Sections R.1–R.3 and Table S66.

2.2 How did we verify answer extraction on the new experiments?

We added independent human verification of 200 outputs generated with a 1,024-token limit. These outputs come from 143 base questions and cover four models, three question sets, and four types of experiments. Two annotators independently read the original responses and recorded their answers without access to the answer keys or parser outputs. A third annotator reviewed all 200 records.

The two annotators agreed on all 200 answer labels and on 199 of the 200 conflict flags. The parser matched the reviewed human labels on 199/200 outputs (99.5%, descriptive Wilson 95% CI 97.22%–99.91%, Cohen's kappa = 0.9935). The only discrepancy was an explicit answer D that the parser missed. Scoring with human and parser extraction yielded 80 and 79 correct outputs, respectively. Both agreed on the 7 outputs that reached the token limit without giving a final choice.

These results support automated answer extraction for scoring this stratified sample. The human check also distinguishes responses with no final answer from responses whose answers the parser missed.

See Section 4.4 and Table 4 of the main text, and Supplementary Section R.4 and Tables S67–S68.

AuthorsOpenReview ↗

Response to Reviewer nVgh——-3-4

3.Can you provide a minimal working input construction, processor invocation traces, and scoring reproduction paths—including accessible versions of missing dependencies, reconstruction methods for frozen issues, and publicly available new raw output samples? You are not required to redistribute restricted images, but you should enable reviewers to verify the end-to-end process from legally obtained upstream data to predicted labels.

The supplementary materials include ICLR2027_Minimal_Reproducibility_11-17.zip, our minimal reproducibility package. The same ZIP is available in the public Hugging Face repository. It includes pinned dependencies, installable fla_core and flash_linear_attention wheels (both version 0.6.0), scripts to reconstruct frozen questions and build inputs, processor invocation traces, and raw responses and generated token IDs from the revised experiments.

The README walks through one fixed VisualMem question (persona_8/event_151/q_1/rotation_0). Reviewers obtain its five images from the upstream dataset under its terms and verify their hashes. They can then rebuild the Native and RPS inputs, run the processor and inspect its traces, generate responses with the recorded model revision, and extract and score the predicted labels.

To replay scoring from saved outputs on CPU, run python reproduce.py verify and python reproduce.py replay --output work/replayed_scores.json. These commands check 208 stored predictions, reported separately as 8 technical examples and 200 sampled outputs from the revised experiments.

See Section 3.3 and the Reproducibility Statement in the main text, and Supplementary Sections R.1–R.3 and Table S66.

4.For recent experiments where many setups still encounter budget exhaustion or unresolved outputs, particularly with external InternVL, do you have additional sufficient output budgets or paired results under controlled answer-output protocols? This would help assess how much of the external RPS benefit can be preserved after improving answer extractability. The focus here is on the interpretability scope of external evidence, rather than demanding uniform conclusions regarding internal mechanisms.

We ran two types of paired diagnostics. Raising the output-token limit increased the number of extractable answers in some settings. Under controlled decoding, all answers were extractable in both groups. RPS still scored higher in both tests, although the evidence is limited.

On the original set of 120 questions constructed from MuirBench, we compared Flat and RPS using the same InternVL3.5-8B-HF model, images, and image order. We raised the limit from 512 to 1,024 and 2,048 tokens. The number of extractable answers increased in both groups. Even at 2,048 tokens, only 46/120 answers were extractable under Flat and 54/120 under RPS. RPS scored higher at all three limits. At 2,048 tokens, the gap was 28.33 percentage points. The Holm-adjusted p-value was 0.1875 for each of the three differences. In the new VisioMath diagnostic with InternVL3.5-14B-HF, the number of extractable answers in the adjacent-neutral condition also rose, from 95/300 at 1,024 tokens to 178/300 at 4,096 tokens.

To compare scores when all answers were extractable, we prefilled FINAL= under controlled decoding. The model then selected a valid option letter, and generation ended. On MuirBench, all 120 outputs in each group had extractable answers. RPS scored 25.83 percentage points higher, a descriptive difference on this constructed question set. In the same diagnostic on VisioMath, all answers were extractable under both Native and RPS. RPS scored 7.33 percentage points higher. The difference was not significant after Holm correction (adjusted p = 0.1912).

The score gaps observed when all answers were extractable show that differences in answer extractability cannot fully explain the results under these controlled conditions. Controlled decoding changes the generation process, so we limit the interpretation of these gaps to the diagnostic settings.

See Sections 4.4 and 4.6 and Table 6 of the main text, and Supplementary Sections P.2–P.3 and Tables S60–S62.

Reviewer_KakwOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper audits multimodal memory evaluation. It argues that the evaluation pipeline of a public benchmark omits bindings between reference and candidate images, and introduces role-preserving serialization (RPS) to fix the issue. The paper tests several RPS designs that differ in mapping placement and wording. It also varies other settings that can affect evaluation, such as the output-token limit, the answer-parsing rules, etc. The paper reports how combinations of these settings affect scores. Finally, it argues that evaluation settings should be specified explicitly so that score differences can be interpreted in terms of retrieval quality.

Critical strengths

The paper identifies a problem in an existing benchmark and explores evaluation settings that have not been examined before. The findings may be of interest to the community.

Critical weaknesses

The main issue is presentation. Terms are often used before they are defined, or are never defined. The rationale for each experiment is insufficiently stated. The Method and Experiments sections are confusing: various experiments are introduced with little explanation, and it does not clearly state what question each comparison answers or why it is needed.

Questions for authors
  • Please make the main text self-contained by clearly defining all key terms and configurations where they first appear.
  • Please explain the intent behind each experimental design where it is introduced, so that readers need not search the paper for the relevant statements.
2 comments
AuthorsOpenReview ↗

Response to Reviewer Kakw——-1

1. Please make the main text self-contained by clearly defining all key terms and configurations where they first appear.

We apologize for the difficulty readers had in reading and understanding the original manuscript. Your comments helped us recognize the need to explain how the terms and configurations relate to one another. We therefore reviewed and manually rewrote the manuscript throughout, adding definitions where the relevant concepts first appear.

Our manuscript examines factors that affect scores in multimodal memory system evaluations. We find that the evaluation interface is an important factor that is often overlooked. We study whether images are correctly bound to their corresponding answer choices, where and how these bindings are stated, how many tokens the system generates, how answers are extracted from the output, and how external reference answers affect the scores. Our experiments address these questions.

We also standardized the terminology throughout the manuscript. Section 3.1 defines role-preserving serialization (RPS) and the original input format, Native, which is called Official in VisualMem. It also explains how answer labels map to images in the input sequence. Section 3.2 introduces the four conditions: global-neutral (GN), global-semantic (GS), adjacent-neutral (AN), and adjacent-semantic (AS). It also introduces Anchor, the control condition that uses the shared prompt template. Section 4.1 defines accuracy, answer parsing coverage, the output-token limit, and the statistical unit so readers can interpret the experimental settings and results directly from the main text. We hope these explanations and the consistent terminology make our study easier to follow.

AuthorsOpenReview ↗

Response to Reviewer Kakw——-2

2. Please explain the intent behind each experimental design where it is introduced, so that readers need not search the paper for the relevant statements.

Your comment about the motivation for the experiments helped us clarify how each comparison relates to our main research questions. We agree that each experiment should state its purpose when it is introduced. We therefore reorganized the methods and experiments around the research questions.

Section 4.1 introduces the three representative public datasets we audited for evaluating memory system performance.

Section 4.2 tests whether adding or strengthening binding statements changes answer accuracy when the retrieved evidence and image order are fixed. It also quantifies how much correct binding changes accuracy.

Section 4.3 keeps all mappings correct and examines the position and wording of the binding statements separately. This helps distinguish the effect of correct binding from the effect of how the binding is presented.

Section 4.4 checks whether models complete their answers, examines how answer extraction rules affect memory performance evaluation, and assesses the accuracy of answer extraction and scoring.

Section 4.5 examines how interface changes affect different retrieval methods.

Section 4.6 uses external reference image injection and controlled answer generation to examine how memory evaluation scores differ across answering conditions.

With these changes, the methods and experiments address three questions in order. Are the bindings explicit? How are the binding statements presented? How do generation and scoring conditions affect the measured results? This order makes the relationship between the comparisons clearer.

Reviewer_Tw3AOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In this draft, the authors audited the VisualMem benchmark and found that reference images and image answer options are given without explicit role or option bindings, which they called a role collision. They proposed role-preserving serialization (RPS) to add these bindings while keeping retrieval fixed. Experiments on 58 image-choice questions showed accuracy improvements of up to 34.9 points on three of four models.

Critical strengths
  1. The main comparison is well controlled, and the wrong-binding control shows that the correct bindings, not the extra text, drive the gain.
  2. The reporting is very transparent, including a human audit of the answer parser.
Critical weaknesses
  1. The scope is very narrow (one benchmark, 58 questions, small open models), and it is not surprising that adding missing option labels helps.
  2. The headline results use a 128-token limit where Qwen often does not finish its answer, so the gain is mixed with truncation.
  3. The downstream claims are not supported: no retrieval comparison passes the correction, and the best retrieval method does not change.
  4. Minor: the main text reads like an experiment log and is hard to follow, with many abbreviations not defined.
  5. Why there is no line number in the draft?
Questions for authors

Please refer to the Critical Weaknesses section for details.

3 comments
AuthorsOpenReview ↗

Response to Reviewer Tw3A—–1

1. The scope is very narrow (one benchmark, 58 questions, small open models), and it is not surprising that adding missing option labels helps.

1.1 Expanded question sets and models

The main binding experiment in the original manuscript used 58 VisualMem questions with image answer choices. The revised manuscript retains these 58 questions and adds fixed subsets of 300 questions each from MMR-Life and VisioMath. This brings the total to 658 base questions. The four cyclic choice orders use the same 58 VisualMem questions, yielding 232 question-order instances. Both new question sets retain their original choice orders.

In the main experiment, we replaced the original Qwen3.5-9B and InternVL3.5-8B-HF with Qwen3.5-27B and InternVL3.5-14B-HF, respectively.

1.2 Does the effect come solely from adding missing labels?

We agree that a score increase from adding missing labels is unsurprising on VisualMem. Of the two new datasets, MMR-Life already places choice letters after the candidate images. VisioMath already provides a global mapping. Among the three question sets, the original VisualMem input lacks explicit labels for the image choices. Across the three datasets, we use the term role collision for an input condition in which image roles or mappings between answer choices and images are insufficiently explicit. To address role collision, role-preserving serialization (RPS) adds explicit image roles and choice labels. It preserves the original mappings, the images, and their order.

Our experiments show that RPS improves accuracy across multiple datasets and models. At a common output limit of 1,024 tokens, 5 of the 12 comparisons of the original input (Native) with RPS show gains that pass Holm correction (Section 4.2 and Table 2 of the main text). Qwen3.5-27B gains 24.3 and 8.3 percentage points on MMR-Life and VisioMath, respectively. InternVL3.5-14B-HF gains 30.2 and 28.3 percentage points on VisualMem and MMR-Life, respectively. Gemma3-4B gains 19.0 percentage points on VisualMem. For Molmo2-8B, the accuracy difference is −1.7 percentage points on VisualMem (Holm-adjusted p = 1.0000). We observe no gains for this model on either new question set. Molmo2-8B illustrates a limitation of our RPS method. We also updated the experiments on mapping-statement position and role wording (Section 4.3, Table 3, and Figure 3 of the main text and Supplementary Section O.2).

The two new question sets broaden the scope of the evaluation. The results show that making image roles and choice mappings more explicit can improve accuracy for some models in both input formats with existing choice labels.

See Sections 3.1 and 4.1–4.2 and Tables 1–2 of the main text. In the supplementary material, see Sections N.1–N.4 and O.1 and Table S55.

AuthorsOpenReview ↗

Response to Reviewer Tw3A—–2-3

  1. The headline results use a 128-token limit where Qwen often does not finish its answer, so the gain is mixed with truncation.

2.1 Raising the main experiment's limit to 1,024 tokens

We agree that a 128-token limit can cut off responses before the model gives an extractable answer. This can affect the original accuracy comparisons. To reduce this problem, the revision sets a common output limit of 1,024 tokens for the main binding comparisons across four models and three question sets. We include every response in the accuracy denominator and count responses without an extractable answer as incorrect. Of 6,656 outputs from the binding experiments, 184 still reach the 1,024-token limit without an extractable answer.

2.2 Testing RPS gains with longer generation budgets

The short budget in the original manuscript amplified the measured gain for Qwen3.5-9B on VisualMem. In a paired comparison with the same questions and inputs, raising the limit from 128 to 512 tokens reduced the gain of RPS over Official from 34.48 to 18.53 percentage points. The reduction was 15.95 percentage points (Holm-adjusted p = 0.01563, Supplementary Section P.2).

The revision also examines the effect of raising the limit from 1,024 to 2,048 tokens. On the same host with matched inputs, the accuracy gain of RPS over Native for Qwen3.5-27B on VisualMem increased by 0.86 percentage points. We did not detect a significant change in the relative gain (Holm-adjusted p = 1.0000). This suggests that 1,024 tokens is a reasonable output limit for Qwen3.5-27B in this setting.

Longer budgets still affect whether answers can be extracted in some conditions. For InternVL3.5-14B-HF on VisioMath with adjacent placement and neutral wording, 95, 130, and 178 answers were extractable at 1,024, 2,048, and 4,096 tokens, respectively, out of 300 at each limit. When the limit rose from 1,024 to 2,048 tokens, the adjacent-minus-global accuracy gap under neutral wording changed by +6.00 percentage points (Holm-adjusted p = 0.00024). At 4,096 tokens, the full Native/RPS comparison yielded a gain of +8.33 percentage points. This gain did not pass Holm correction (adjusted p = 0.0535). In the settings we tested, longer generation budgets made more answers extractable under some conditions and changed the gaps in some placement comparisons (Supplementary Sections P.1–P.2 and Tables S60–S61).

The gains reported at 128 tokens in the original manuscript should therefore be interpreted in light of truncation. The 1,024-token limit provides a common basis for comparison in the revision. Answer extractability and some placement comparisons remain sensitive to the generation budget.

See Sections 4.1, 4.4, and 4.6 and Tables 2 and 6 of the main text. In the supplementary material, see Sections P.1–P.2 and Tables S55 and S60–S61.

  1. The downstream claims are not supported: no retrieval comparison passes the correction, and the best retrieval method does not change

We agree that after correction for multiple comparisons, the evidence is insufficient to determine whether RPS changes which retrieval configuration performs best. Section 4.5 of the revised manuscript limits this analysis to how serialization affects measured accuracy gaps. The comparison uses 58 VisualMem questions with image answer choices in their original choice order and a common output limit of 1,024 tokens. Within each configuration (Naive, Retrieval-only, and Full), the original input and RPS use the same retrieved evidence.

For Gemma3-4B, Full answers 1 fewer question correctly than Naive under the original input and 11 more under RPS. The accuracy gap between the two configurations shifts by 20.7 percentage points (Holm-adjusted p = 0.25). None of the 4 primary or 8 secondary interaction comparisons passes its respective Holm correction.

The ranking change for Gemma3-4B is a descriptive result on these 58 questions. Comparisons of retrieval configurations should state which serialization was used. Determining which configuration performs better requires further validation. This is a limitation of our RPS method in retrieval comparisons.

See Section 4.5 of the main text (p. 8, lines 429–431, and p. 9, lines 470–474), as well as Table 5, Figure 4, and Equation 4. See also Supplementary Section O.3 and Table S59.

AuthorsOpenReview ↗

Response to Reviewer Tw3A—–4-5

  1. Minor: the main text reads like an experiment log and is hard to follow, with many abbreviations not defined.

Your comments helped us revisit the manuscript from the reader’s perspective and explain more clearly why we conducted each experiment and how the experiments relate to one another. In response, we revised the order of presentation in the main text and added and standardized definitions of key terms.

First, we organized the methods and experiments around our research questions. We now explain why we conduct each set of comparisons when we introduce it. For example, we compare the original input with RPS to test how adding or strengthening image bindings changes answer scores. In the 2 × 2 comparison, we keep the mappings between images and answer choices correct and examine the position of the mapping statements and the role wording separately. We then examine how output length and answer extraction affect scoring. Next, we test whether changes to the interface affect the score gaps between retrieval methods. This sequence helps readers understand how each experiment addresses the research questions.

Second, we explain the key terms and abbreviations and use them consistently throughout the manuscript. RPS stands for role-preserving serialization. Native refers to the original input, which is called Official in VisualMem. GN, GS, AN, and AS denote the global-neutral, global-semantic, adjacent-neutral, and adjacent-semantic conditions, respectively. We also explain that Anchor is the reference condition that uses the shared prompt template with the added mapping statements and grouping removed. In the experimental setup, we define accuracy, the answer extraction rate, and the output-token limit.

Finally, we now state the purpose and significance of each experiment at the start of its section. We hope these changes make the manuscript easier to understand.

Revised sections:Sections 1, 3, and 4 of the main text and Supplementary Sections N–R.

  1. Why there is no line number in the draft?

The revised anonymous PDF has continuous line numbers. The main text is numbered from line 000 to line 539.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.