acceptodds
Under review as a conference paper at ICLR 2027

True Record, Wrong Subject: Diagnosing Visual Memory Misbinding in Personalized Multimodal Models

Abstract

A personal record can be true without applying to the subject shown in a query. We study this visual memory misbinding through a controlled diagnosis of record applicability and stage-specific decisions. A record-level contract separates subject presence (), image–record assignment (), and answer support (). RecordAuth-Diag contains 3,690 cases with matched assignment interventions and distinct support, presence, and ambiguity controls. Card removal and nonce relabeling establish dependence on supplied records among cases with unsafe outputs before authorization. On 560 localized DAVIS cases, an ATRA-based context guard increases correct record answers from 170 to 228 while reducing unsafe record use from 149 to 5. Filtering full-bank answers by the same retained event set preserves only 144 correct answers. Context selection therefore changes which record answers the generator produces, beyond withholding an existing answer. Retaining the correct target does not ensure that generation uses it, and exact field execution can expose an invalid selected association. A lower rate of unsafe record use in model outputs does not necessarily imply a less contaminated selected context. These interactions motivate a structured-query reference pipeline separating record selection, field execution, and candidate acceptance. Under unchanged checks, the post-hoc Field comparison yields more correct answers and more unsafe releases than checked MLLM readout. Code and reproducibility materials are available at https://anonymous.4open.science/r/visual-memory-misbinding-1E03.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
multimodal large language models, multimodal personalization, visual memory misbinding, record-level authorization

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_eDKkOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper studies record misbinding, where valid infromation is incorrectly linked to a subject. It also introduces a controlled benchmark which seperates subject presence, assignment validtiy and task support. A multi stage pipeline combines record level verification with proposed ATRA, context guard and answer checks. Experiments show ta reduction in unauthorized record use as well as transfer to unseen datasets

Critical strengths

RecordAuth-Diag is a diagnostic dataset which isolates misbindings from other failure types. Work identifies a possibly interesting failure mode, and provides a structured method of detecting and handling such issues. Evaluation framework seperates out different states and helps explainability, evaluating where the system failed.

Critical weaknesses

The distinct contribution is insufficiently clear. The paper combines retrieval, similarity-based verification, rejection rules, and answer checking, but does not clearly establish what new method these components introduce. ATRA’s advantage over the matched Flat-MLP is also encoder-dependent, with no clear improvement on SigLIP2. The diagnostic benchmark is useful, but the broader methodological contribution needs stronger justification.

The presentation makes the work difficult to assess. Dense terminology, undefined acronyms, and frequent changes in context make it difficult to follow the work. This affects more than readability: it is difficult to trace how P/E/S determines each decision and which comparisons support each claim. A worked example and a clearer mapping between conditions, safeguards, and outcomes are needed.

Questions for authors
  1. Contribution clarification: What is the main new insight beyond combining record matching, rejection, and support checks? How do P/E/S and the controlled interventions differ from existing attribution and entity-resolution formulations?
  2. Method clarification: What benefit does ATRA provide over the matched Flat-MLP, given the lack of a clear advantage with SigLIP2? Which existing results isolate its contribution from the deterministic guard and answer checks?
  3. Pipeline clarification: Could one concrete example trace a record through P/E/S, selection, readout, and checking, including the conditions that produce NULL? This would help establish exactly what the system verifies and what remains outside its guarantees.
5 comments
AuthorsOpenReview ↗

Supplementary Response to Reviewer eDKk

Thank you for the time and effort you have devoted to reviewing our paper. Based on our reading of your review, the central concern about our method is what ATRA's learned architecture contributes and how its contribution can be isolated from matching features, deterministic safeguards, and answer checks. Our initial response explained the set-feedback architecture and compared ATRA with MLP under shared downstream processing. To further identify the effective computation, we have added a direct intervention on the learned group- and bank-feedback paths.

A direct intervention on the learned feedback paths.

We evaluate the trained ATRA selector on all 3,690 held-out RecordAuth-Diag cases, from 369 parents and 87 identities, using both encoders and the original five folds and five seeds. All 24 per-card features, learned local card representations, group/NULL scores, weights, standardizers, thresholds, and latest-record readout are held fixed. We measure selector outputs before the deterministic triad guard and candidate-answer checks.

We replace the case-specific learned group context, bank context, or both at the card-decision head with activation means computed from the original seen training conditions in each fold. This directly tests the role of the current record set's learned representations in individual card decisions. The intact implementation exactly reproduces the saved answers, selected groups, and retained sets for both encoders.

Cells show task accuracy / unauthorized-record exposure / coverage (%), each over all 3,690 cases. Accuracy includes correct NULLs; exposure means that a selected set contains an ineligible record, and coverage means a non-NULL selection.

Card-head contextSigLIP2: Acc / Exp / CovDINOv2: Acc / Exp / Cov
Intact ATRA feedback80.46 / 6.53 / 65.7568.29 / 9.73 / 60.62
Group feedback → training mean31.73 / 0.03 / 1.9236.40 / 0.60 / 8.35
Bank feedback → training mean65.04 / 24.55 / 66.8055.18 / 24.66 / 59.97
Both feedback paths → training means39.95 / 0.08 / 11.3044.07 / 1.08 / 18.18

Bank feedback improves the discrimination of ineligible records. With intact bank feedback, exposure is 6.53%/9.73% for SigLIP2/DINOv2, compared with 24.55%/24.66% after mean replacement, while coverage differs by only 1.06/0.65 percentage points. The intact feedback therefore filters substantially more ineligible records at similar coverage. The exposure reductions are 18.02/14.93 points, with identity-clustered 95% intervals of [16.71, 19.30]/[13.28, 16.51]. This supports the role of bank-specific context in distinguishing applicable records from competing evidence.

Group feedback supports useful record retention. Replacing group feedback reduces coverage from 65.75%/60.62% to 1.92%/8.35%, accompanied by a substantial accuracy decrease. The intact group representation thus helps the trained selector admit records that support correct answers. Together, the two interventions connect learned set context to both useful retention and the filtering of ineligible records.

These results identify a concrete contribution of ATRA's hierarchical design: learned representations of the current group and memory bank feed back into record-level authorization, helping retain useful records and reject ineligible ones. The effect is observable with the same relational inputs and group scores, before downstream safeguards, providing direct evidence that learned set feedback contributes to reliable memory use.

Thank you to the authors for the detailed response and the extra experiments. The worked example and Algorithm 1 help a lot. The Bob/Alice walkthrough makes it much easier to follow how P/E/S leads to each decision and when the system returns NULL. The feedback-path intervention also helps, since it measures ATRA before the guard and answer checks. I hope these changes are included in the main manuscript as well for easier readability.

I still have two main concerns.

First, the gain from ATRA over the matched MLP is small. With SigLIP2, the pipeline gain is 2.14 points on DAVIS and 1.94 on VOT. The response says SigLIP2 features may leave fewer ambiguous matches. This is an assumption, and no experiment supports it. The mean-replacement test shows that the trained ATRA relies on its feedback paths. It does not show that these paths beat a model trained without them, which is the MLP comparison. Replacing group feedback also drops coverage from 65.75% to 1.92%. This looks more like the model being pushed outside its training range than proof that group feedback helps.

Second, much of the safety seems to come from the fixed rules. Before the guards, the ATRA selector still exposes ineligible records in 6.53% / 9.73% of cases. The final unsafe rate is about 1%. The algorithm also requires ATRA to agree with Triad and keeps only the records both select. In the walkthrough, the Kyoto record is simply assumed to be excluded. It does not show how ATRA rejects it, and that is the core of the method.

The benchmark is useful, but the method's new contribution is still unclear to me. I will keep my score.

AuthorsOpenReview ↗

Further response to eDKk

Thank you for taking the time to read our revision and replies, and for continuing the discussion. Our earlier response did not fully answer your concerns. Your feedback has helped us clarify the contribution itself, which we value beyond the score. We appreciate this and welcome further discussion.

Our reading of your main concern is: does ATRA's learned feedback add value over a simpler trained selector, and does that value exist before the fixed safeguards? We previously used a feedback intervention to explain the mechanism. You are right that this does not replace a trained comparison. We now connect that comparison to pre-guard results and actual record decisions.

1. Does training with feedback help?

We trained a new control without group/bank feedback to the card head. It keeps the same 24 relational features and group/NULL branch. A wider local head matches capacity: 55,415 parameters versus ATRA's 55,427. Both start from scratch with the same five identity-disjoint folds, five seeds, losses and optimizer, and their own validation-calibrated thresholds. There is no DAVIS/VOT tuning.

Both use the same active/image-present masks and Field readout, before Triad intersection and candidate checks. Cells show accuracy / ineligible-record exposure / non-NULL coverage (%); accuracy includes correct NULLs.

SigLIP2 test setRetrained no-feedbackRetrained ATRA
DAVIS, n=56079.11 / 10.18 / 67.8683.04 / 9.29 / 70.36
VOT, n=31083.23 / 10.97 / 72.5885.16 / 7.74 / 70.97

On DAVIS, accuracy improves by 3.93 points, with correct non-NULL answers increasing from 316 to 338. On VOT, unsafe answers fall from 34 to 24, while correct non-NULL answers rise from 187 to 192. This directly tests learning with feedback, replacing our earlier conjecture about SigLIP2 ambiguity.

2. Where does the improvement enter the pipeline?

We next return to the original locked ATRA and Flat-MLP models and give both the same Field readout, before Triad intersection and candidate checks. This is a new analysis of their saved predictions, separate from the retraining above. Accuracy includes correct NULLs; the other columns are counts.

SourceSelectorAccuracyCorrect non-NULLUnsafeAll non-NULL
DAVISFlat-MLP77.68%31764393
DAVISATRA82.86%33652390
VOTFlat-MLP80.32%18239226
VOTATRA85.48%19323220

ATRA is 5.18/5.16 points ahead before safeguards, with 19/11 more correct non-NULL answers and 12/16 fewer unsafe ones. The final-pipeline gaps are 2.14/1.94 points. Separating these stages shows that ATRA improves the records supplied to the common executor, while the fixed checks further screen its answers.

3. Which mistakes does this avoid?

To replace the assumed Kyoto exclusion with actual decisions, here are two logged cases. Scores come from the representative runs selected by the original ensemble rule; both models choose the correct subject group.

  • Rejecting a misassigned record (DAVIS, bear::1). January 8's Topaz image belongs to rhino::1; January 7's Indigo remains valid. ATRA scores Topaz 0.6136, below its 0.9081 threshold, and retains Indigo at 0.9987. MLP retains Topaz (0.9157, threshold 0.5000). The same latest-record rule therefore returns Indigo for ATRA and Topaz for MLP. The checks accept ATRA's correct answer but turn MLP's answer into NULL.
  • Keeping valid support (VOT, hand). January 5's Blue Station image is copied into another group, but the original record remains valid. At the same 0.5000 threshold, ATRA retains it (0.9474), while MLP drops it (0.4991). The same readout returns Blue Station for ATRA but the older January 4 Hazel Garden for MLP. Both pass the checks: only ATRA answers correctly.

Across all 56 DAVIS / 31 VOT queries, a new clean-to-one-way-misbinding analysis also finds that 13/8 previously correct queries become wrong or abstained with MLP, versus 4/1 with ATRA, using the same final readout and checks.

Together, training with feedback improves selection; that improvement is visible before fixed safeguards; and it helps reject misleading records while keeping valid support that MLP misses. This is the contribution we want to make clearer. Thank you again for helping us explain it better.

Reviewer_nyyXOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The authors study visual memory misbinding in personalized multimodal models: a system may retrieve a fact that is valid for one subject but incorrectly apply it to another. The authors introduce a record-level authorization formulation that distinguishes subject presence (P), the validity of image–record assignment (E), and answer support (S). The paper contributes RecordAuth-Diag, a benchmark containing 3,690 controlled cases constructed from 369 parent examples and 87 identities, along with assignment, support, presence, and ambiguity interventions. It introduces an Anchor-Triadic Relational Authorizer (ATRA) that combines learned record-level selection with deterministic visual consistency checks. On DAVIS-560, the ATRA context guard improves correct record answers from 170 to 228 while reducing unsafe record use from 149 to 5.

Critical strengths
  • The paper identifies a failure mode distinct from ordinary hallucination: a model can retrieve a genuine memory record yet apply it to the wrong visual subject. The P/E/S authorization formulation helps clarify this distinction. Reciprocal assignment swaps preserve the record contents while altering associations, and the memory-removal and nonce-relabeling experiments provide evidence that the observed failures involve supplied records rather than merely generic answer hallucination (Table 1).
  • The diagnostic design is well controled. Reciprocal swaps keep record text and the image multiset fixed while changing only the assignment. Together with memory-bank removal and nonce relabeling, this shows that the errors come from reusing supplied records.
Critical weaknesses
  • The evaluation remains narrow relative to the proposed general problem of visual memory authorization. Most experiments use controlled identity associations and synthetic event metadata, with a heavily structured latest-location query schema. Evaluation sets are small (56 DAVIS confirmation tracks, not video-disjoint; 310 VOT cases), and utility gains depend on the source: MyVLM clean accuracy falls from 51.26% to 25.21%, and Qwen positive recall drops slightly. Because of this, it's unclear whether the proposed pipeline is a general contribution or a solution tuned to this diagnostic. The authors acknowledge that natural error prevalence is unmeasured and free-form evaluation lacks human validation (Section 7).
  • The method's practical utility and architectural advantage are not consistently established. In Table 3, ATRA's advantage over a similarly sized Flat-MLP is encoder-dependent: with SigLIP2, Flat-MLP slightly exceeds ATRA in native accuracy (81.25% versus 80.46%), while their unsafe exposure rates are similar (6.91% versus 6.53%). Intersecting ATRA with the triad rule lowers exposure but substantially reduces coverage and positive recall.
Questions for authors
  • Can the authors provide a coverage- or clean-accuracy-matched comparison between ATRA, Flat-MLP, and the deterministic anchor-triad baseline?
  • How should the negative MyVLM transfer results affect the intended scope of the method? In particular, can the authors identify which differences between RecordAuth-Diag, DAVIS/VOT, and MyVLM account for the large clean-accuracy loss?
  • How much do the DAVIS development and confirmation tracks overlap in source videos, and how do the main DAVIS numbers change on a video-disjoint subset?
  • Can the Field + check pipeline extend beyond latest-location queries to free-form questions, and how would S be checked there?
3 comments
AuthorsOpenReview ↗

Supplementary Response to Reviewer nyyX

Thank you for the time and effort you have devoted to reviewing our paper. Based on our reading of your review, the central concern about our method is whether record authorization transfers from structured diagnostics to free-form personalization with gains in both usefulness and reliability. Our initial response reported Omni-Persona transfer with two generators using the fixed authorizer and described free-form generation and support checking. To further clarify how these gains arise, we now provide a finer outcome analysis of the answerable questions.

Outcome decomposition on answerable free-form questions.

We analyze the 10% misbinding condition: 115 unique answerable questions evaluated under three corruption seeds, giving 345 question-seed evaluations per configuration. The authorizer retains its original weights and thresholds. Within each generator family, the MLP and ATRA variants share triad, generation, and checking. We manually verified the counts of correct answers, incorrect released answers, and abstentions.

ConfigurationCorrectIncorrect, releasedAbstainedCorrect / released
Qwen1644014180.39%
Qwen + MLP1742514687.44%
Qwen + ATRA186915095.38%
InternVL1604913676.56%
InternVL + MLP1713613882.61%
InternVL + ATRA1831914390.59%

The three outcome counts in each row sum to 345. The final column measures correctness among released answers in this answerable subset: correct / (correct + incorrect released).

Relative to the base generators, ATRA produces 22/23 more correct answers and 31/30 fewer incorrect released answers for Qwen/InternVL, with abstentions increasing by 9/7. Relative to MLP, it produces 12 more correct answers for each generator, alongside 16/17 fewer incorrect released answers and 4/5 more abstentions. The improvement therefore combines an increase in successful answers with a reduction in erroneous responses.

The same pattern holds for the quality of released answers: correctness rises from 80.39% to 95.38% for Qwen and from 76.56% to 90.59% for InternVL. ATRA thus delivers both more correct answers and a higher proportion of correct responses among the answers it releases.

With generation and checking shared between the learned-selector variants, the advantage over MLP connects these gains to record selection. The fixed ATRA authorizer helps both generators use personalized memories more effectively, supporting the transfer of record-level authorization from structured readout to free-form answer generation.

Dear Reviewer nyyX, thank you again for taking the time to review our paper. Your question about going beyond latest-location lookup prompted the Omni-Persona comparison in the first table of our initial reply. At 10% added misbinding, both Qwen and InternVL gave more correct free-form answers with ATRA than with Flat-MLP, without retuning ATRA. If you have a moment to look over our response and the revised manuscript, we’d welcome your thoughts on that comparison.

Thank you authors for the detailed responses and the additional evidence! My concerns are largely addressed by the additional evidence, although the gains of ATRA over the matched MLP are still unclear (similar to eDKk's comment), especially given the limited size of the evaluation set. what are the paired 95% Confidence Intervals of (Qwen/InternVL + ATRA) - (Qwen/InternVL + MLP) on the different metrics?

Reviewer_v3ByOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This work studies visual memory misbinding in which a personalized MLLM answers with a true record that belongs to a different subject when a image-record assignment is corrupted in its memory. The work proposes a framework to test this problem by decoupling subject presence (P), assignment validity (E) and task support (S). The work also proposes a benchmark with 3690 cases across ten assignment and support conditions.

Critical strengths
  1. The problem statement is relevant and current VLMs are prone to the problem of visual misbinding.

  2. The formulation of experiments is controlled which leads to meaningful findings across different task conditions.

Critical weaknesses
  1. Presentation is extremely poor and paper is hard to understand.

    • The main paper acts like an index to a mammoth 45-page appendix and the reader has to refer to the appendix to read a lot of details.
    • The introduction and general wording of the entire paper seems extremely entangled. For example, the authors show the example in Figure-1 but they do not refer to the same example while defining the terms in Section-3's beginning. The paper would become clearer if the authors can describe the terms in Equation-1 using example in Figure-1.
    • In the introduction itself, I can point out the following issues: (i) L042-045 - The authors start using terms such as "clean bank", "record", "image-record assignment", that are not defined before. (ii) L047-052 - Again terms such as "task's recency rule", "right group", "card", used as standard terms used and not defined. (iii) Figure 1 caption - terms P, E and S used but it is not clear what they mean.
    • There are LLM-like constructs throughout the entire text for example "A not B" kind of statements, e.g. Sec. 3.1, L139: “The index j describes where a card is supplied, not whether that assignment is valid.”; Sec. 3.2, L174: treating the anchor “as visual evidence rather than authenticated provenance.”, etc.
  2. The paper's central claim is to fix image reference misbinding but most of the gains are from recency execution. 77 of the guard-MLLM's eligible errors are older-record errors with the target retained (Table 7).

  3. ATRA's value add to this work is not well established. Table 52 from the appendix shows that a capacity-matched MLP is almost as good using the SigLIP2 encoder. On held out intervention families, ATRA is less accurate than the parameter-free anchor-triad rule (Table 34 - 79.77 vs 74.07).

Questions for authors

In addition to answering to the points mentioned in weakness 2 and 3, I would like the authors to answer the following questions:

  1. Can the DAVIS and VOT results be split by intervention families seen versus unseen during ATRA training?

  2. What are the end-to-end correct/unsafe trade-offs at realistic and low-corruption prevalence?

1 comment
AuthorsOpenReview ↗

Supplementary Response to Reviewer v3By

Thank you for the time and effort you have devoted to reviewing our paper. Based on our reading of your review, the central concern about our method is whether record authorization itself provides additional benefits beyond the latest-record readout rule, and where ATRA offers advantages over MLP and triad. Our initial response provided fixed-readout controls, seen/unseen-family and low-corruption results, and a clearer method description. To address this concern further, we now provide finer selector-level and query-level analyses of those experiments.

1. ATRA improves record selection under shared readout and checks.

We decompose the DAVIS/VOT results by intervention families seen or unseen during fitting/calibration. The MLP and ATRA pipelines use the same triad intersection, Field readout, and candidate checks; the triad-only selector also uses the same readout and checks. Weights and operating thresholds remain fixed. All rows below contain answerable tasks, and cells give correct / unsafe answer counts.

SourceFamiliesnTriad + readout/checksMLP pipelineATRA pipeline
DAVISseen11266 / 385 / 184 / 1
DAVISunseen280200 / 3192 / 1205 / 1
VOTseen6246 / 252 / 052 / 0
VOTunseen155130 / 2122 / 2128 / 0

On unseen intervention families, ATRA gives 13/6 more correct answers than MLP on DAVIS/VOT, with unsafe answers unchanged at 1 on DAVIS and reduced from 2 to 0 on VOT. The paired analysis identifies 20/11 cases where ATRA answers correctly and MLP does not, compared with 7/5 in the reverse direction. These paired results show that ATRA's selection benefit carries over to interventions absent from fitting/calibration, enabling the same readout to produce more correct answers.

The one-way assignment corruption illustrates this benefit directly. Among 56 DAVIS cases, replacing MLP with ATRA changes correct/unsafe counts from 32/1 to 41/1; among 31 VOT cases, the change is 19/2 to 26/0. Since the latest-record rule and checks are identical, these gains locate ATRA's contribution in the records supplied to execution. Correctly applying the recency rule requires selecting applicable records first, and ATRA improves this preceding step.

Across all answerable tasks in the table, adding ATRA to the triad-based pipeline yields 23/4 more correct answers on DAVIS/VOT while reducing unsafe answers from 6/4 to 2/0. Thus, learned selection complements the deterministic consistency rule by improving both useful answering and filtering of inapplicable records.

2. The complete pipeline also improves useful answering without added misbinding.

At zero added misbinding, we compare Direct answering with the complete ATRA pipeline on the same 56 DAVIS and 31 VOT queries, after deduplicating repeated allocations. “Additional correct answers” counts queries that Direct answering did not answer correctly but the pipeline did. “Previously correct, now abstained” counts queries that Direct answering answered correctly but the pipeline abstained on. Their difference gives the net gain in correct answers.

SourceUnique queriesAdditional correct answersPreviously correct, now abstainedNet gain in correct answers
DAVIS56149+5
VOT3191+8

On both sources, the additional correct answers outnumber the previously correct answers that become abstentions, yielding 5/8 net additional correct answers. The complete pipeline therefore provides a positive answer-utility gain even without injected misbinding, showing that its selection and readout stages work together to use available memories more effectively.

Together, these analyses connect ATRA's design to two concrete benefits: better record selection with readout held fixed, and more correct answers from the complete pipeline on clean queries. They substantiate the value of authorizing records before executing the task's readout rule.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.