acceptodds
Withdrawn from ICLR 2027

Models Already Know Where to Look: Recovering the Discarded Spatial Map in Parallel Scanpath Prediction

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Task-driven scanpath prediction models an ordered, variable-length sequence of fixations made during a visual task. Some parallel models supervise a spatial distribution for each fixation slot but discard it at inference, using an independent coordinate-regression head instead. We study this representation–decoding mismatch through a controlled family of models. C0 uses coordinate regression, S0 adds an auxiliary spatial map, and S0-SpatialDecode (S0-SD) decodes the map of an already trained S0 checkpoint without retraining. Task-Conditioned Spatial Decoding (TCSD) instead trains and predicts through the same spatial distribution. On an image-disjoint COCO-Search18 evaluation, S0-SD improves all six metrics over S0 under joint seed–image resampling. A stronger regression head narrows but does not close the gap to spatial decoding. S0-SD and TCSD show no reliable difference on any metric, indicating that most of the gain comes from using the learned map at inference rather than from the TCSD training pathway. The effect persists with ImageNet-pretrained features, while experiments on Gazeformer and AiR-D show that its transfer is metric- and task-dependent. TCSD retains efficient one-pass inference, although HAT-SF remains more accurate on most metrics. These results show that spatial supervision and the decoder that turns the learned map into fixation coordinates should be evaluated together.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Eye Tracking, Visual Scanpath Prediction, Parallel Scanpath Prediction, Representation–Decoding Mismatch, Spatial Decoding

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_FJjAOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

The paper studies the problem of task-driven, parallel scanpath modelling, in models that use spatial fixation maps. The key question is how the spatial distribution for each fixation should be used to predict scanpath locations at training and inference. The core contribution is a set of analyses with a controlled set of models that re-use the same shared representations but differ in the way fixation coordinates are regressed or sampled. What they show is that by better using the spatial distribution during inference much of the gap with more complex models (like their RCSD).

However, at its current state, my overall assessment is that of a paper lacking both writing and experimental depth, and as such not meeting the bar for acceptance at ICLR: the contributions and motivations are not clear, the presentation is quite shallow and repetitive at times, the model names are hard to follow through the text, and while nicely controlled, the key experiments are quite simple in nature and performed only on the authors’ models, e.g. without trying to retrieve the same mechanism from the existing baselines.

Critical strengths
  1. The experiments comparing their model classes are clean and well-isolated.
  2. The evaluation protocol is careful, also with respect to image splits and avoidance of test-set leakage, and with re-training of the external baseline and SOTA models.
Critical weaknesses
  1. The core contribution is unclear, and the writing itself does not help in highlighting it. The main result that the author show, i.e. that using spatial supervision in scanpath prediction is better than a regression head to coordinates that bypasses it, seems un-surprising. Models like DeepGaze and some of the reported baselines already operate similarly. While it is helpful to see the effect, this is shown only for the authors’ models, which are not SOTA on any metric. Does this affect also other models and could be used to improve them?

  2. Despite the simplicity of its results, I found the paper quite hard to follow. Results are reported but not deeply interpreted. The central claims are restated multiple times in the abstract, introduction, discussion and conclusion but they are never fully fleshed out in terms of applications, consequences, effects on other models or impact to the broader field. The space used to repeat the claims could have been used e.g. to better motivate the paper, give definition for the metrics (for which also citations are missing), better interpret the results, and so on. The naming of the models is also initially quite hard to follow, as multiple models are present and similarly named. This is mostly evident in the abstract, which I think could and should probably be re-written to convey the main message of the paper without naming the different abbreviations several times. In comparing the models, and helping keeping track, a single table mapping each variant to its training losses, inference pathway and inputs would have helped a lot.

A more minor point is that the paper references an appendix, which I cannot see in the current submission pdf.

Questions for authors

See above point 1.

Reviewer_LytVOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The authors investigate a representation–decoding mismatch in parallel scanpath prediction which stems from the regression head's tendency to miss crucial localization information in the supervised spatial map. For this purpose they introduce a controlled family of parallel fixation predictors: C0, S0, S0-SD, and finally TCSD which is the authors claim to be an end-to-end aligned implementation of spatial decoding. Through experiments and ablations on target-present subset of COCO-Search18, the authors show that S0-SD models consistently improves over S0 despite not adding any additional trained parameter, and also recovers most of the gain achieved by the TCSD pathway.

Critical strengths
  1. Systematic analysis of scanpath decoding: The authors constructed a systematic framework to investigate the pitfalls of parallel scanpath decoding methods through several experiments and ablations - shedding light on the various mechanisms underlying scanpath decoding.
Critical weaknesses
  1. Main text references an apparently absent appendix: The main text refers to an appendix for critical details but I couldn't find it.
  2. The proposed TCSD method is not SOTA: The proposed TCSD method is suboptimal - faring even worse than Gazeformer on NSS and SED, and much worse than HAT-SF.
  3. The text could have been better-structured: The motivation and main findings are not well-highlighted in the text - I found it hard to grasp the main message of the paper which I believe got lost in text.
Questions for authors
  1. Can the authors comment on why one should prefer TCSD to S0-SD?
  2. Can the authors provide details meant to be in the appendix?
Reviewer_JC4AOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents a work on task-driven scanpath prediction. The main contributions include identifying a representation–decoding mismatch in parallel scanpath prediction, task-conditioned spatial decoding trains and predicts through a single spatial representation. Experiments validate the proposed method with detailed discussions.

Critical strengths

The paper is well presented. The main contributions are clear presented and analyzed. Experiments are provided with some detailed discussions.

Critical weaknesses

The proposed TCSD does not show clear advance compared to existing methods, especially HAT-SF. The authors also mentioned that the compared models are retrained but do not provide details of the retraining to show if the retrained models are optimal. The paper missed a lot of closely related work, especially those papers discussing scanpath predictions or gaze predictions. It is necessary to provide a more thorough review of related work on visual search, scanpath or gaze predictions, so that the claimed contributions can be clearly distinguished from existing approaches.

Questions for authors

The proposed TCSD does not show clear advance compared to existing methods, especially HAT-SF. The authors also mentioned that the compared models were retrained but did not provide details of the retraining to show if the retrained models are optimal.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.