acceptodds
Under review as a conference paper at ICLR 2027

PRISM: Predicting Complete Scene-Memory Report Distributions from Visual Geometry

Abstract

Visual working memory preserves information about a scene after it is no longer visible. Yet it is usually modeled through accuracy or error magnitude, which discard which scene a person reports. Two equally distant errors can differ in visual content, making error identity a richer probe of remembered representations. We introduce PRISM, a framework for predicting complete distributions of scene reports from target–candidate visual similarity. PRISM-Eval isolates what frozen vision networks contribute beyond response-display structure and six fixed image descriptors. PRISM-Channel places visual evidence within a cognitive model of memory precision and guessing, while Cognitive-Channel Geometry Distillation (CCGD) learns a compact combination of frozen similarities. Across two continuous-report datasets, intermediate visual features improve held-out likelihood beyond all fixed controls. Combining three visual models improves cognitive-baseline predictions, with relative-weight gains under fixed regularization and positive adapted transfer to an independent memory task. Independent EEG data show that the behaviorally useful geometry captures neural structure beyond six fixed descriptors. PRISM thus turns memory error identity into a controlled bridge between natural-scene vision, working-memory theory, and neural representations.

Primary area:
applications to neuroscience & cognitive science
Keywords:
visual working memory, pretrained vision models, cognitive modeling, supervised learning, brain–model alignment

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_F2TJOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper examines whether visual similarity predicts the full distribution of scene-memory reports beyond response position and fixed image descriptors. PRISM evaluates frozen vision models, adds visual evidence to a memory model, and tests a learned combination of visual features. Intermediate features improve held-out predictions in two behavioral datasets, and independent EEG data show related representational structure. An adapted two-choice transfer test is positive, although the primary nested 360-choice tests do not establish an advantage for learned unequal weights.

Critical strengths

(1) With representation selection confined to the training folds, ViT stage 3 improves held-out prediction beyond response topology and six jointly fitted image descriptors in both datasets (+0.02897 and +0.01916 nats/trial; Sec. 5.2, Fig. 3). This is the strongest evidence that the frozen representation contributes useful information beyond the fixed controls. (2) PRISM-Channel improves on the cognitive baseline, and the gain disappears when visual evidence is matched to the wrong target (Sec. 5.4, Fig. 4). This supports a target-specific contribution to predicting report identity.

Critical weaknesses

(1) The primary nested comparison does not establish that learned CCGD weights outperform a fitted equal-weight ensemble in either dataset (Sec. 5.5; App. G.6). The small positive results with rho = 1 are exploratory: this setting was used during development on both archives, and the later nested analysis does not undo earlier specification choices (App. G.2). The bootstrap intervals resample retained out-of-fold scores rather than the historical model search (App. B.5). My concern is about the strength of the separate weight-learning claim; I do not see evidence of leakage in the held-out evaluation. (2) The full 360-choice evaluations cover three routes and five wheels (Sec. 4.1). The independent memory test shows positive adapted transfer, but fits task-specific components for a two-choice task rather than transferring the complete 360-choice predictor unchanged (App. K.4). This limits generalization, without indicating a technical error in the held-out tests. These concerns temper my overall recommendation, while the held-out ViT and Channel results remain persuasive.

Questions for authors

(1) Requested analysis: Under the opposite-half cross-fitting protocol used for Fig. 4, how does PRISM-Channel compare on the same Scene Wheels test halves with a cognitive baseline augmented by the six fixed image descriptors? This would clarify whether the Channel gain extends beyond those descriptors. (2) Clarification: Appendix G.2 notes that rho = 1 was used during development on both behavioral archives. Were any other CCGD specification choices made after examining those archives? If so, which ones?

4 comments
AuthorsOpenReview ↗

Response to F2TJ (1/2): Matched Channel comparison

Thank you for distinguishing the supported representation result from the separate relative-weight claim, and for requesting the matched Channel control. We completed that analysis and report it in Section 5.4 and Appendix L. It does not establish a Channel advantage over a cognitive baseline augmented with the six fixed descriptors. We retain this result and make the hierarchy explicit: controlled representation evaluation is the principal contribution; the primary nested tests do not establish learned-weight superiority.

Reply to Q1 — Matched opposite-half Channel comparison

We reused Figure 4's Scene Wheels rows, 20 observers, ordered 360-candidate galleries, balanced trial halves, and observer-by-radius cognitive baseline. Both directions are evaluated, so each of the 5,400 responses is scored once. All calibration and descriptor fitting use the opposite half; no test response selects a setting.

The descriptor comparator is q6 ∝ q exp(βᵀx6): six signed coefficients per observer, shared across radii. Ridge is selected from {0.01, 0.1, 1} by three target-disjoint inner folds within the calibration half. The cognitive baseline is refitted on each inner training fold before scoring validation targets. The six HOG/Gabor definitions and image-only preprocessing follow the submitted specification. Original Channel retains its submitted temperature/mixing grid and training-half selection; common-inner-selected variants are reported separately. This matches data and cognitive calibration, without claiming equal parameter counts.

Original observer-P50 Channel minus observer-specific nested q6 gives +0.001666 nats/trial, pointwise 95% participant-bootstrap CI [−0.008987, 0.012610]. The two-sided exact participant sign-flip p is 0.77068, and Holm-adjusted p is 1.0 across all 18 declared all-report/far-report comparisons. Corresponding NLLs are 4.388774 for Channel and 4.390440 for q6. Against the original cognitive baseline, q6 gains +0.035905 and Channel +0.037571. Thus the descriptor comparator achieves most of the original increment; the direct comparison establishes neither Channel superiority nor equivalence.

The sensitivities fixed in the post-review protocol (written before these fits, though not an independent preregistration) give the same all-report conclusion. Selecting Channel on the same inner-validation rows gives −0.000775 versus observer-specific q6, CI [−0.011487, 0.010672]. Original Channel versus pooled, shared-coefficient q6 gives −0.000407 [−0.008981, 0.008234]. Removing P50 or fixing descriptor ridge at 1 also does not establish an advantage.

We report the unfavorable sensitivity explicitly: on the common original-baseline P50 far-report subset, original Channel is worse than shared q6 by −0.030777, CI [−0.047667, −0.013628], Holm18 p 0.03939. Unconstrained q6 can change total tail mass, whereas P50 holds that mass fixed; this comparison therefore concerns far-report prediction, not solely redistribution within the tail. Intervals are pointwise and conditional on fitted procedures and sampled wheels, not uncertainty over historical specification search or new environments.

This is the requested opposite-half analysis; the earlier environment-held-out cognitive-offset comparison cannot substitute for it. The positive PRISM-Eval increment beyond topology and six jointly fitted descriptors addresses a different calibration/generalization question and does not overturn this Channel result.

AuthorsOpenReview ↗

Response to F2TJ (2/2): Development history and generalization

Reply to W1 & Q2 — Learned weights and development history

Yes: the submitted configuration documents choices beyond fixing rho = 1. These include restricting predictors to the three stage-3 representations, defining the A Walk conditional baseline (Appendix J.1), a three-stage3-kernel conditional-ridge sweep over {0, 10⁻⁵, 10⁻⁴, 10⁻³, 0.01, 0.1, 1}, a twelve-layer conditional-ridge variant at {0.1, 1, 10}, and nested ridge selection on A Walk. It records choosing fixed ridge 1 for stable cross-dataset environment transfer and direct neural portability. Appendix G.2 now makes these documented searches explicit.

Both behavioral archives informed development. The surviving configuration is not an exhaustive, timestamped history, and we cannot reliably reconstruct every undocumented decision. We therefore do not claim that rho was the only outcome-informed choice. “Neural portability” describes the recorded rationale; the reported EEG coefficients were fitted to behavior, not EEG.

The later nested procedure controls the choices it actually implements; it does not erase earlier development. Retained-score bootstrap intervals do not replay that search. The primary nested CCGD-versus-fitted-equal comparisons do not establish unequal-weight superiority, and fixed-rho increments remain exploratory. This concerns claim strength, separate from the held-out evaluation's validity, as your review emphasized.

Reply to W2 — Generalization and adapted transfer

Appendix P adds frozen PRISM-Eval transfer (under a retrospective protocol fixed before scoring) to the original archive's ISC2 practice gallery. Selection and all 197 readout fits use only the unchanged 15,035 main-route reports; no practice response recalibrates the predictor. Selected ViT stage 3 plus six controls improves over six controls by +0.077344 nats/trial, crossed 95% CI [0.048373, 0.104933], on all 1,005 valid reports. The retained-source 978-report sensitivity gives +0.078926 using the same predictions.

This adds one 360-position response gallery, but observed targets occupy only positions 0–89 and 110 of 118 sessions overlap training. It supports narrow additional-gallery transfer, not independent-cohort or broad environment confirmation. The separate two-choice test remains adapted weight-direction transfer with a fitted task-specific baseline and scalar strength, not unchanged transfer of the complete 360-choice predictor, and its gain over equal weighting (+0.000600 nats/trial) does not establish superiority over every single encoder (descriptive NLL 0.480844 vs. 0.480623 for ViT alone; Appendix K.4).

AuthorsOpenReview ↗

Follow-up on our rebuttal

Dear Reviewer F2TJ,

Thank you again for your time and thoughtful feedback. As the discussion period draws to a close, we would appreciate your assessment of whether our responses and the revised manuscript address your concerns. We would be happy to clarify any remaining questions.

Thank you for considering our revisions.

Best regards, The authors

Thank you for the detailed response and additional analyses, including the Channel comparison and the clarification of the model development process. I still find the representation analysis valuable, especially because adding visual representations improved predictions in both datasets after accounting for the layout of the response options and the six image descriptors. However, the additional results do not establish that PRISM-Channel performs better than the cognitive baseline augmented with image descriptors, or that learned CCGD weights perform better than equal weighting. The additional transfer test covers a limited range of images, and the training and testing sessions overlap. Overall, I think the controlled representation results are sufficient to support weak acceptance, so I will maintain my original rating.

Reviewer_C32POpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes PRISM, a framework that leverages representational similarities from pretrained vision models to predict complete probability distributions over 360 candidate scenes in human visual working memory tasks. The framework consists of three components: PRISM-Eval evaluates the additional predictive value of deep visual representations by controlling for response topology and conventional visual features; PRISM-Channel incorporates visual similarity into traditional cognitive memory models through probabilistic reweighting to predict specific memory errors; and CCGD learns combination weights over similarities from three frozen visual encoders.

The authors evaluate their approach on two datasets, A Walk in the Park and Scene Wheels, and further examine the correspondence between visual representations and neural activity using an independent EEG dataset. The results suggest that intermediate ViT features provide predictive information beyond conventional visual descriptors, while combining multiple visual models also improves the prediction of human memory responses.

Critical strengths

1. Rigorous experimental controls and representation analysis.

The paper systematically controls for response topology, color, edge, and texture information to isolate the contribution of deep visual representations. The finding that intermediate ViT features provide additional predictive value beyond six fixed visual descriptors is supported by carefully designed experiments.

2. Comprehensive empirical validation.

The paper evaluates complete response distributions across two datasets, with held-out environment tests, ablations, and independent EEG analysis. These experiments provide credible evidence for the relationship between visual similarity and human memory responses.

Critical weaknesses

1. Insufficiently established significance beyond human behavioral prediction.

While modeling human cognition is scientifically valuable, the paper does not clearly demonstrate what new insights or capabilities its improved behavioral predictions enable, particularly in the context of an AI conference such as ICLR.

Related works provide clearer contributions beyond behavioral fitting:

  • DreamSim (Fu et al., NeurIPS 2023) [1] learns a human-aligned perceptual metric with demonstrated applications in image retrieval and reconstruction.
  • RTify (Cheng et al., NeurIPS 2024) [2] introduces a framework that aligns neural network decision dynamics with human reaction times and supports speed–accuracy optimization.
  • Setting Up for Failure (Radmard et al., ICLR 2026) [3] analyzes the internal dynamics of RNNs trained to reproduce memory errors and proposes testable neural hypotheses.

In contrast, PRISM mainly demonstrates improvements in predictive NLL and correlations with EEG representations. It remains unclear how these findings inform AI representation learning or model design, or what new, testable insights they provide into cognitive mechanisms. This limits the broader significance of the contribution.

2. Limited methodological novelty and insufficient evidence for the necessity of key components.

Bates et al. (2024) [4] already use pretrained visual similarities and the TCC framework to predict natural-scene memory errors, including on the same Scene Wheels dataset. PRISM therefore builds on an established modeling paradigm.

Its main methodological additions also raise concerns:

  • PRISM-Channel relies primarily on relatively simple probabilistic reweighting. Removing the P50 constraint leaves overall predictive performance nearly unchanged.
  • CCGD learns only three similarity coefficients and does not significantly outperform equal-weight ensembling under the primary nested model-selection protocol.

Although the stronger experimental controls are valuable, these results do not sufficiently establish the necessity or methodological novelty of the proposed components.

Questions for authors

1. Broader significance and contributions to AI research.

Beyond improving the prediction of human memory responses, what new methodological or mechanistic insights does PRISM provide? In particular, can the finding that intermediate ViT features are more predictive than final-layer features lead to actionable guidance for visual representation learning, model evaluation, or testable hypotheses about memory mechanisms?

2. Necessity and novelty of the proposed components.

Given that prior work already predicts complete memory-error distributions using deep visual similarities [4], what are the key advances of PRISM-Channel and CCGD? Specifically, how do the authors justify the value of P50 and learned similarity weights when their removal or replacement with simpler alternatives yields comparable overall predictive performance?

References

[1] Fu, S., Tamir, N. Y., Sundaram, S., Chai, L., Zhang, R., Dekel, T., & Isola, P. (2023). DreamSim: Learning New Dimensions of Human Visual Similarity Using Synthetic Data. Advances in Neural Information Processing Systems (NeurIPS 2023), 36, 50742–50768. https://proceedings.neurips.cc/paper_files/paper/2023/hash/9f09f316a3eaf59d9ced5ffaefe97e0f-Abstract-Conference.html

[2] Cheng, Y.-A., Rodriguez, I. F., Chen, S., Kar, K., Watanabe, T., & Serre, T. (2024). RTify: Aligning Deep Neural Networks with Human Behavioral Decisions. Advances in Neural Information Processing Systems (NeurIPS 2024), 37, 130485–130510. https://proceedings.neurips.cc/paper_files/paper/2024/hash/eb696eb3c461f631389b4b61a8465794-Abstract-Conference.html

[3] Radmard, P., Bays, P. M., & Lengyel, M. (2026). Setting Up for Failure: Automatic Discovery of the Neural Mechanisms of Cognitive Errors. International Conference on Learning Representations (ICLR 2026). https://proceedings.iclr.cc/paper_files/paper/2026/hash/45eaa675af0e4c600852768d9b65a132-Abstract-Conference.html

[4] Bates, C. J., Alvarez, G. A., & Gershman, S. J. (2024). Scaling Models of Visual Working Memory to Natural Images. Communications Psychology, 2, Article 3. https://doi.org/10.1038/s44271-023-00048-3

4 comments
AuthorsOpenReview ↗

Response to C32P (1/2): Broader significance

Thank you for asking what these controlled findings enable. PRISM tests which visual information predicts memory-report identity beyond response layout and conventional image cues. Intermediate representations can contain useful information that pooled embeddings obscure, while more elaborate readouts need not improve on simpler alternatives. Section 6 and Appendix N now develop the theoretical interpretation and prospective tests. The concluding paragraph of Section 6 sets out the broader research aim.

Reply to W1 & Q1 — Broader significance and testable implications

Bates et al. (2024) already predict complete scene-memory reports with deep similarities and TCC, compare layers, and include pixel, color, and autoencoder baselines. Building on that work, PRISM tests the additional contribution after response topology and six fixed descriptors are fitted jointly, with representation selection within the declared retrospective candidate family confined to development environments.

The resulting distinction matters empirically. On Scene Wheels, HOG outperforms every tested pooled pretrained representation, yet adding the selected intermediate ViT representation beyond six jointly fitted descriptors improves held-out likelihood by +0.01916 nats/trial; the corresponding A Walk gain is +0.02897 (Section 5.2). Thus apparent alignment of a pooled representation and additional information in an internal representation yield different conclusions. Our recommendation is to compare internal layers under common likelihoods, joint visual controls, and held-out selection, while testing simple ensemble alternatives.

The additional ISC2 gallery test supports this controlled increment without practice recalibration: +0.077344 nats/trial, 95% CI [0.048373, 0.104933] (Appendix P). Selection and fitting use only the original main routes. This is one same-study practice gallery with restricted targets and overlapping sessions, rather than independent-cohort confirmation. The matched Qwen comparison also finds no established language-backbone advantage over its selected image tower; the Scene Wheels difference is negative under six primary comparisons but spans zero under the broader all-18 correction (Appendix O). These results favor evaluating the representation and readout actually used, rather than assuming that a later or more elaborate representation will improve memory prediction.

DreamSim, RTify, and Radmard et al. demonstrate, respectively, perceptual-metric applications, temporal decision modeling, and candidate error-producing dynamics. PRISM contributes a complementary, controlled evaluation of representation information in memory choices. Its current evidence supports this use, without claiming demonstrated improvements in general visual learning or the downstream applications of those works.

AuthorsOpenReview ↗

Response to C32P (2/2): Cognitive interpretation and model choices

Cognitive interpretation and a prospective test.

Channel models how visual evidence changes report probabilities. This could reflect readout of an uncertain memory or confusability in the maintained representation; either can produce the same report distribution. The behavioral model observes neither trialwise subjective uncertainty nor the latent memory state, and its fitted visual strength is not an uncertainty-gated strategy. P50's outer region denotes candidate outcomes under the baseline, not low-confidence trials.

There is positive motivation for investigating this distinction. Yoo et al. (2021) show that people maintain item-specific uncertainty and use it in memory decisions; Li et al. (2021) link decoded spatial-memory content and uncertainty to trialwise reports. These findings motivate testing scene content and decision demands jointly; they do not overturn our matched descriptor-control result.

TCC already includes a decision rule. Brady & Robinson (2026) use TCC-Luce/softmax over measured similarities for their main 360-choice analyses; Gaussian noisy maximum is another TCC implementation. Channel instead applies a visual odds adjustment to a separately calibrated response baseline, using signed-offset-centered similarity, followed by optional P50 protection and baseline mixing. This is a distinction between model constructions, not storage versus decision or competition versus no competition: the unshielded proposal's softmax admits a noisy-maximum interpretation (Robinson et al., 2023).

A prospective test would use distance- and descriptor-matched scenes with differing model predictions, then vary response options or late-cued demands after identical encoding. One model would hold independently measured memory geometry fixed and vary its readout; another would predict reports from measured geometry changes. Decoding before and after the cue, with uncertainty and response time, could test these restrictions. Shan et al. (2025) report different neural-bias time courses in orientation recall, motivating this timing test; joint choice–time modeling adds constraints (Fennell & Ratcliff, 2023). These tests remain prospective: a late cue or null decoder result cannot establish a pure decision mechanism. Appendix K.5 recovers only stipulated simulated rivals, and current EEG correspondence does not localize a causal stage. PRISM supplies image-computable predictions for these tests.

Reply to W2 & Q2 — Novelty, design choices, and simpler alternatives

The exponential tilt and conditional-logit fit specialize established statistical tools; the controlled comparisons are the advance. P50 is an optional diagnostic constraint: it preserves each central candidate probability and total outer mass, while permitting redistribution outside. It does not preserve the entire error-magnitude distribution. Gains with and without P50 are +0.03757 and +0.03756 nats/trial, so it is unnecessary for the aggregate improvement. The new opposite-half comparison does not establish Channel superiority over the cognitive baseline augmented with six descriptors: +0.001666, 95% CI [−0.008987, 0.012610] (Appendix L; shared-coefficient sensitivities also favor the descriptor model on the far-error subset: −0.030777, Holm18 p = 0.03939). We retain this result separately from the positive environment-held-out PRISM-Eval comparison.

CCGD provides a compact conditional readout and a controlled test of combining similarities versus learning their relative weights. Its primary nested gains over fitted equal weighting are +0.000516 and −0.002267 nats/trial; both intervals include zero (Appendix G.6). Thus unequal weighting is not established as necessary or superior. Fixed-ridge increments remain exploratory due to archive-informed specification development; independent two-choice transfer uses a frozen weight direction with a new baseline and scalar (Appendix K.4). The value is an explicit, testable comparison of representational information and readout choices, without requiring every optional component to improve prediction.

Thank you for the detailed response and additional experiments. I acknowledge the value of the controlled representation analysis, but my main concerns remain. While cognitive modeling is scientifically meaningful, I find it more compelling for an AI conference when it provides actionable insights for improving AI models or advances our understanding of cognitive mechanisms. PRISM's contributions remain primarily focused on behavioral prediction and representation evaluation, while its proposed mechanistic implications have not been empirically validated. Moreover, the additional results do not establish clear advantages for PRISM-Channel over the six-descriptor baseline, CCGD over equal-weight ensembling, or P50 over its removal. Although these components may serve as useful analytical tools, their independent methodological contributions remain limited. Therefore, I will maintain my original rating.

AuthorsOpenReview ↗

Clarification of scope and theoretical contribution

Thank you for carefully considering our response and additional experiments, and for clarifying your remaining concerns. We appreciate the opportunity to clarify the scope and theoretical contribution of our work! Different combinations of memory and decision processes can generate the same report distribution, so predictive fit and EEG correspondence alone do not identify a unique mechanism. PRISM tests which candidate visual representations predict additional structure in human scene-memory confusions after response topology and measured image descriptors are controlled. These comparisons constrain representational assumptions without attributing the errors to a particular maintenance or readout process.

The theoretical contribution concerns how representational similarity enters accounts of visual working memory limits. Descriptions of item capacity or average precision do not, by themselves, specify which natural scenes will be confused. PRISM provides behavioral and neural evidence for evaluating candidate similarity structures beyond the measured visual controls. This supplies quantitative constraints on the representational assumptions within capacity- versus noise-based theories: they must account for the identities and relative probabilities of errors alongside overall accuracy and error spread. Section 6 and Appendix N outline late-cue and model-disagreement experiments that could further distinguish maintenance and readout accounts.

The main PRISM-Eval analysis does not use PRISM-Channel, CCGD, or P50 (Sections 3.1 and 3.2). Adding development-selected intermediate ViT features to response topology and six jointly fitted descriptors improves trial-weighted held-out log likelihood by 0.02897 nats/trial on A Walk and 0.01916 on Scene Wheels; both 97.5% per-dataset confidence intervals exclude zero (Section 5.2; Appendix C.2, Table 4). ViT stage-3 geometry also shows positive partial representational similarity with delay-averaged EEG dissimilarities from independent participants and scenes after controlling all six descriptors (Section 5.6; Appendix K.2).

These findings provide concrete guidance for selecting representations in computational models of human memory. On Scene Wheels, HOG yields a higher mean held-out likelihood gain over response topology than every tested pooled pretrained representation, yet intermediate ViT features add information beyond the joint descriptor baseline (Sections 5.1–5.2). In the matched Qwen2.5-VL-3B-Instruct comparison, neither tested language-backbone readout establishes an advantage over the selected visual tower (Appendix O). Thus, pooled embeddings and language states should be compared with internal visual layers under shared likelihoods, visual controls, and held-out selection when evaluating human scene-confusion geometry.

The component comparisons support narrower claims. Channel has no established advantage over the matched descriptor-augmented cognitive baseline (Appendix L), and nested selection does not establish a CCGD advantage over fitted equal weighting (Section 5.5; Appendix G.6). Removing P50 leaves the aggregate predictive gain nearly unchanged in the Scene Wheels ablation (Figure 4). P50 nevertheless enables a specific diagnostic comparison by preserving each central probability and total outer mass while allowing redistribution among outer-region candidates (Appendix N).

Reviewer_GEYUOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper introduces PRISM, which is a framework for predicting human scene memory choices using behavioral datasets. Given a target image and candidate images, it predicts the probability that a person will select each candidate after a memory delay. Using image similarity feature from vision encoders, PRISM-Eval compares representations, PRISM-Channel adds visual similarity, and CCGD learns a combination of frozen similarities. The main result suggests that intermediate ViT features improve prediction beyond response distance and simple image descriptors.

Critical strengths
  • Useful empirical findings. The paper shares a novel finding that intermediate ViT features improve the prediction across multiple behavioral datasets.

  • Interesting research question. Understanding the mechanisms of visual working memory of humans and their relevance of AI models is an important research question in the era of human-like AI agents.

Critical weaknesses
  • Unclear motivation and objective. I find the motivation of this research insufficiently justified in the current manuscript. Is the main goal to better explain human's visual working memory, evaluate difference visual representations, or develop a better predictive model? The paper should explain why improved prediction matters for that goal and why/how the main experiments are addressing it.

  • Low readability and incoherent structure. The motivation, methods, and analyses do not form a clear and coherent narrative. For instance, PRISM-Eval, PRISM-Channel, CCGD, EEG comparisons and other transfer experiments are presented without a clear hierarchy of research questions and contributions. Given this, it is hard to connect each experiments to the central objective and understand which findings are actually important.

  • Insufficient visual examples and qualitative analysis. The only visual examples are the small Scene Wheels illustration in Figure 1. Providing representative exmaples from the used datasets, showing the target, candidate scenes, human selection, model propobabilities, etc. would help explain which visual differences drive human confusions and where the models capture or miss them. With the presentation, the analysis on visual working memory is not straightforward.

  • Unclear main takeaway and significance for computer vision research. The major results seems to be that ViT features add predictive information beyond simple image descriptors. However, the discussion doesn't make broader implications. What does this finding reveal about human memory or visual representations, and how should it inform computer vision or foundation model research? I'm curious about the expected implications of the findings in the broder literature of computer vision, machine learning, or AI in general.

  • Comparison with multimodal LLMs (MLLMs). Given that the paper is aiming to relate model representations to human scene memory, I'm curious why the authors didn't consider multimodal LLMs as a comparison rather than the vision encoders. The experiments include vision encoders like CLIP, but do not conduct similar experiments on MLLMs that use these encoders. A similar experiment using the internal representations in the language backbone of MLLMs could provide useful insights on whether processing beyond the vision encoders improves prediction of human memory errors. Without this comparison, the relevance of the findings to multimodal foundation models remains unclear.

Questions for authors
  • Are there any qualitative patterns in cases where the model predicts human outputs well and poorly? Could you provide representative successes and failures from both datasets to clarify which scene properties or types of memory errors the model captures?

  • How should Figure 3-C be interpreted? S3 seems to be is selected in every environment.

  • What should the core takeaway from the paper be? What do the results teach us about human visual working memory or models' visual representations overall?

4 comments
AuthorsOpenReview ↗

Response to GEYU (1/2): Objective, structure, and examples

Thank you for identifying the need for a clearer objective, an understandable hierarchy, concrete examples, and a comparison within an MLLM. We have revised the presentation around one question: which visual representations predict the particular scene a person reports from memory, beyond response layout and measured image structure? Section 5.3 now details the cognitive-model comparison, and Section 6 closes with the broader research aim.

Reply to W1 & W4 & Q3 — Objective and broader significance

A scene-memory error contains more information than whether the answer was correct or how far it fell from the target. Two candidate scenes can be equally distant in the response display yet differ in visual content and in how often people choose them. Predicting the probability of each candidate tests whether a representation captures this difference. Held-out likelihood rewards a model for assigning more probability to the responses people actually make; the controls test whether this information extends beyond simpler explanations.

For representation research, the important distinction is between apparent alignment and additional predictive information. On Scene Wheels, HOG outperforms every tested pooled pretrained representation, yet an internal ViT representation adds information beyond six jointly fitted HOG/Gabor descriptors. Thus a pooled embedding alone can give an incomplete account of a network's usefulness for this task. The selected-representation gains are +0.02897 nats/trial on A Walk and +0.01916 on Scene Wheels. This motivates evaluating internal layers under shared likelihoods, strong image controls, and held-out selection. For cognition, this asks which scene distinctions shape confusions after accounting for error spread and measured image cues. Independent EEG correspondence provides a converging test of representational structure; it does not locate the effect in storage or report selection.

Reply to W2 — Organization and hierarchy of contributions

The revised Introduction and Results give PRISM-Eval this central role. Channel examines target-matched evidence within a calibrated memory baseline; CCGD fits one coefficient for each of three frozen encoder similarities, separating the benefit of combining representations from learning their relative weights. EEG and transfer provide additional tests with their own scope. Neither the new Channel-versus-descriptor comparison nor nested CCGD-versus-equal weighting establishes superiority. Their role no longer depends on treating every component as a necessary methodological advance.

Reply to W3 & Q1 — Qualitative successes and failures

Appendix M presents four held-out error trials, one relative gain and one loss from each dataset, with target/report images, probabilities, and complete distribution curves. The A Walk examples show nearby park views with changes in viewpoint and tree positions; the Scene Wheels examples show a bright dining interior and a blue-green bedroom. They illustrate local visual ambiguity without establishing that a particular object or scene category causes the model's gain or loss.

Adding the selected representation changes the observed-report probability from 2.0404% to 2.2800% in the A Walk gain example and 3.3456% to 3.0665% in its loss example. The Scene Wheels changes are 1.9113% to 2.0892% and 5.0114% to 4.6101%. All four distributions remain modal at the target; even the improved reports rank only 16th and 14th among 360 candidates. These are improvements in probability assignment, not exact report recovery.

We selected each trial nearest its dataset-and-gain-sign group's median gain before viewing images. This retrospective, outcome-conditioned selection illustrates both directions, not their prevalence. The aggregate counts supply that context: likelihood improves on 57.4% of incorrect A Walk reports and 52.6% of incorrect Scene Wheels reports. Appendix M also reports target-level counts; 3,310 of 4,295 observed Scene Wheels targets have only one report. Consequently, neither these individual responses nor the examples establish scene-specific human response distributions.

AuthorsOpenReview ↗

Response to GEYU (2/2): Layer selection and MLLM comparison

Reply to Q2 — Interpretation of Figure 3C

S3 is the ViT-B/16 CLS representation after transformer block 9. It was selected in all three route and five wheel tests. Within each test, five target-disjoint inner folds compare twelve architecture-stage candidates and three ridge values using only the other environments, before refitting and scoring the excluded environment. The panel documents repeated selection within this declared retrospective family. Development sets overlap, and pooled outcomes and the globally nominated S3 were already known during development; these are not eight independent discoveries of a universally optimal depth.

Reply to W5 — Matched MLLM comparison

Under a retrospective protocol fixed before fitting to behavior, new Figure 3D and Appendix O compare internal representations of frozen Qwen2.5-VL-3B-Instruct: four tower depths, the merger, four image-token language depths, and four assistant-prefix depths. The same images and fixed prompt use the submitted five-inner-fold, environment-held-out evaluation and six descriptor controls; no encoder is trained.

Image-token language states minus the selected tower yield −0.000835 nats/trial on A Walk, simultaneous primary-six interval [−0.007106, 0.006687], and −0.010917 on Scene Wheels, [−0.021940, −0.000412]. Neither establishes a language advantage. The Scene Wheels difference favors the tower under the six primary comparisons, but its broader all-18 interval spans zero. In A Walk, the intermediate merger improves over the six-descriptor baseline (+0.009938) and over image-token language states (+0.004349), while the assistant-prefix readout underperforms the tower on Scene Wheels (−0.012842); all selections and all eighteen comparisons are retained.

The combined findings support testing internal visual states for memory prediction; additional language-backbone processing shows no demonstrated increment over its matched tower under these readouts. This conclusion is specific to one checkpoint, prompt, and pooling scheme. It neither ranks all MLLMs nor isolates a causal role for linguistic semantics.

AuthorsOpenReview ↗

Follow-up on our rebuttal

Dear Reviewer GEYU,

Thank you again for your time and thoughtful feedback. As the discussion period draws to a close, we would appreciate your assessment of whether our responses and the revised manuscript address your concerns. We would be happy to clarify any remaining questions.

Thank you for considering our revisions.

Best regards, The authors

Thank you for the detailed response with clarifications and additional experiments. The qualitative examples in Section M helped understand the exact task being addressed in the paper, and the comparison with MLLMs is also helpful. Given these changes, I have raised my rating to 2.

However, the fundamental motivation and practical significance of this research remains unclear to me, making it difficult to further increase my rating.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.