- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper examines whether visual similarity predicts the full distribution of scene-memory reports beyond response position and fixed image descriptors. PRISM evaluates frozen vision models, adds visual evidence to a memory model, and tests a learned combination of visual features. Intermediate features improve held-out predictions in two behavioral datasets, and independent EEG data show related representational structure. An adapted two-choice transfer test is positive, although the primary nested 360-choice tests do not establish an advantage for learned unequal weights.
Critical strengths
(1) With representation selection confined to the training folds, ViT stage 3 improves held-out prediction beyond response topology and six jointly fitted image descriptors in both datasets (+0.02897 and +0.01916 nats/trial; Sec. 5.2, Fig. 3). This is the strongest evidence that the frozen representation contributes useful information beyond the fixed controls. (2) PRISM-Channel improves on the cognitive baseline, and the gain disappears when visual evidence is matched to the wrong target (Sec. 5.4, Fig. 4). This supports a target-specific contribution to predicting report identity.
Critical weaknesses
(1) The primary nested comparison does not establish that learned CCGD weights outperform a fitted equal-weight ensemble in either dataset (Sec. 5.5; App. G.6). The small positive results with rho = 1 are exploratory: this setting was used during development on both archives, and the later nested analysis does not undo earlier specification choices (App. G.2). The bootstrap intervals resample retained out-of-fold scores rather than the historical model search (App. B.5). My concern is about the strength of the separate weight-learning claim; I do not see evidence of leakage in the held-out evaluation. (2) The full 360-choice evaluations cover three routes and five wheels (Sec. 4.1). The independent memory test shows positive adapted transfer, but fits task-specific components for a two-choice task rather than transferring the complete 360-choice predictor unchanged (App. K.4). This limits generalization, without indicating a technical error in the held-out tests. These concerns temper my overall recommendation, while the held-out ViT and Channel results remain persuasive.
Questions for authors
(1) Requested analysis: Under the opposite-half cross-fitting protocol used for Fig. 4, how does PRISM-Channel compare on the same Scene Wheels test halves with a cognitive baseline augmented by the six fixed image descriptors? This would clarify whether the Channel gain extends beyond those descriptors. (2) Clarification: Appendix G.2 notes that rho = 1 was used during development on both behavioral archives. Were any other CCGD specification choices made after examining those archives? If so, which ones?
4 comments
Response to F2TJ (2/2): Development history and generalization
Reply to W1 & Q2 — Learned weights and development history
Yes: the submitted configuration documents choices beyond fixing rho = 1. These include restricting predictors to the three stage-3 representations, defining the A Walk conditional baseline (Appendix J.1), a three-stage3-kernel conditional-ridge sweep over {0, 10⁻⁵, 10⁻⁴, 10⁻³, 0.01, 0.1, 1}, a twelve-layer conditional-ridge variant at {0.1, 1, 10}, and nested ridge selection on A Walk. It records choosing fixed ridge 1 for stable cross-dataset environment transfer and direct neural portability. Appendix G.2 now makes these documented searches explicit.
Both behavioral archives informed development. The surviving configuration is not an exhaustive, timestamped history, and we cannot reliably reconstruct every undocumented decision. We therefore do not claim that rho was the only outcome-informed choice. “Neural portability” describes the recorded rationale; the reported EEG coefficients were fitted to behavior, not EEG.
The later nested procedure controls the choices it actually implements; it does not erase earlier development. Retained-score bootstrap intervals do not replay that search. The primary nested CCGD-versus-fitted-equal comparisons do not establish unequal-weight superiority, and fixed-rho increments remain exploratory. This concerns claim strength, separate from the held-out evaluation's validity, as your review emphasized.
Reply to W2 — Generalization and adapted transfer
Appendix P adds frozen PRISM-Eval transfer (under a retrospective protocol fixed before scoring) to the original archive's ISC2 practice gallery. Selection and all 197 readout fits use only the unchanged 15,035 main-route reports; no practice response recalibrates the predictor. Selected ViT stage 3 plus six controls improves over six controls by +0.077344 nats/trial, crossed 95% CI [0.048373, 0.104933], on all 1,005 valid reports. The retained-source 978-report sensitivity gives +0.078926 using the same predictions.
This adds one 360-position response gallery, but observed targets occupy only positions 0–89 and 110 of 118 sessions overlap training. It supports narrow additional-gallery transfer, not independent-cohort or broad environment confirmation. The separate two-choice test remains adapted weight-direction transfer with a fitted task-specific baseline and scalar strength, not unchanged transfer of the complete 360-choice predictor, and its gain over equal weighting (+0.000600 nats/trial) does not establish superiority over every single encoder (descriptive NLL 0.480844 vs. 0.480623 for ViT alone; Appendix K.4).
Follow-up on our rebuttal
Dear Reviewer F2TJ,
Thank you again for your time and thoughtful feedback. As the discussion period draws to a close, we would appreciate your assessment of whether our responses and the revised manuscript address your concerns. We would be happy to clarify any remaining questions.
Thank you for considering our revisions.
Best regards, The authors
Thank you for the detailed response and additional analyses, including the Channel comparison and the clarification of the model development process. I still find the representation analysis valuable, especially because adding visual representations improved predictions in both datasets after accounting for the layout of the response options and the six image descriptors. However, the additional results do not establish that PRISM-Channel performs better than the cognitive baseline augmented with image descriptors, or that learned CCGD weights perform better than equal weighting. The additional transfer test covers a limited range of images, and the training and testing sessions overlap. Overall, I think the controlled representation results are sufficient to support weak acceptance, so I will maintain my original rating.
Response to F2TJ (1/2): Matched Channel comparison
Thank you for distinguishing the supported representation result from the separate relative-weight claim, and for requesting the matched Channel control. We completed that analysis and report it in Section 5.4 and Appendix L. It does not establish a Channel advantage over a cognitive baseline augmented with the six fixed descriptors. We retain this result and make the hierarchy explicit: controlled representation evaluation is the principal contribution; the primary nested tests do not establish learned-weight superiority.
Reply to Q1 — Matched opposite-half Channel comparison
We reused Figure 4's Scene Wheels rows, 20 observers, ordered 360-candidate galleries, balanced trial halves, and observer-by-radius cognitive baseline. Both directions are evaluated, so each of the 5,400 responses is scored once. All calibration and descriptor fitting use the opposite half; no test response selects a setting.
The descriptor comparator is
q6 ∝ q exp(βᵀx6): six signed coefficients per observer, shared across radii. Ridge is selected from {0.01, 0.1, 1} by three target-disjoint inner folds within the calibration half. The cognitive baseline is refitted on each inner training fold before scoring validation targets. The six HOG/Gabor definitions and image-only preprocessing follow the submitted specification. Original Channel retains its submitted temperature/mixing grid and training-half selection; common-inner-selected variants are reported separately. This matches data and cognitive calibration, without claiming equal parameter counts.Original observer-P50 Channel minus observer-specific nested q6 gives +0.001666 nats/trial, pointwise 95% participant-bootstrap CI [−0.008987, 0.012610]. The two-sided exact participant sign-flip p is 0.77068, and Holm-adjusted p is 1.0 across all 18 declared all-report/far-report comparisons. Corresponding NLLs are 4.388774 for Channel and 4.390440 for q6. Against the original cognitive baseline, q6 gains +0.035905 and Channel +0.037571. Thus the descriptor comparator achieves most of the original increment; the direct comparison establishes neither Channel superiority nor equivalence.
The sensitivities fixed in the post-review protocol (written before these fits, though not an independent preregistration) give the same all-report conclusion. Selecting Channel on the same inner-validation rows gives −0.000775 versus observer-specific q6, CI [−0.011487, 0.010672]. Original Channel versus pooled, shared-coefficient q6 gives −0.000407 [−0.008981, 0.008234]. Removing P50 or fixing descriptor ridge at 1 also does not establish an advantage.
We report the unfavorable sensitivity explicitly: on the common original-baseline P50 far-report subset, original Channel is worse than shared q6 by −0.030777, CI [−0.047667, −0.013628], Holm18 p 0.03939. Unconstrained q6 can change total tail mass, whereas P50 holds that mass fixed; this comparison therefore concerns far-report prediction, not solely redistribution within the tail. Intervals are pointwise and conditional on fitted procedures and sampled wheels, not uncertainty over historical specification search or new environments.
This is the requested opposite-half analysis; the earlier environment-held-out cognitive-offset comparison cannot substitute for it. The positive PRISM-Eval increment beyond topology and six jointly fitted descriptors addresses a different calibration/generalization question and does not overturn this Channel result.