- Rating
- 3
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper carries out work involving saliency modelling, specifically in the case where there is a temporal domain to the assessment. The paper suggests that there is considerable room for improvement still in establishing visual saliency when the content driving it is temporal in nature. SalTempo is introduced as a new visual benchmark suitable for this kind of assessment. This includes 224 action-centred, one minute videos including raw eye tracking, training/evaluation splits, and a benchmark of 13 model variants. Much emphasis focuses on the SalTempo model and analysis reveals that unlike the existing LEDOV evaluation, SalTempo reveals greater ability to assess saliency specific to motion information. This results in a nice dataset/benchmark contribution in allowing stronger future assessment of temporal saliency as well as a current snapshot of model behaviour.
Critical strengths
- This is undoubtedly a useful dataset given the number of video samples and also very high sampling rate to the gaze data itself. Some of the additional metadata that allows assessment specific to an individual observer, and the long sample length and density allows for most kinds of assessment one might want to perform.
- The dataset is also strong in many of the controls included such as repeated frames, and assessment of different attributes that might contribute to a ceiling on performance. Overall this is a solid dataset contribution. A meaningful distinction from LEDOV is also established through assessments that are rigorous and well motivated.
Critical weaknesses
- The analysis is generally good, but there are some places where the claims go a bit further than what the evidence supports. In particular, low cross-frame density establishes changing gaze locations, but doesn't necessarily guarantee that prediction requires history. Therefore, this isn't yet strong enough to state that tracking is required.
Questions for authors
- Can you distinguish explicitly between a model's measured dependence on the temporal context of frames and information unavailable to a trained static predictor? i.e. temporal context alone doesn't always guarantee that there is no information a static predictor can see.
- Can you provide additional detail on gaze-to-frame processing and eye selection, the probability normalization used for model outputs, adaptation settings, and checkpoint selection?
3 comments
Continued
We use the following evaluation pipeline:
The 500Hz recordings are binocular and the released files keep both eyes. Our analyses use the left eye only, with no averaging across eyes. Missing and off-screen samples are dropped. Each sample carries a fractional video-frame index and is assigned to frame when , which gives about 270 samples per frame with up to 16 observers on each evaluation video. Scoring is at the native . All frames with gaze are scored, including the center-anchored first frames.
Models that output a saliency map, including SalFoM, have the map resized bilinearly to the scoring resolution and divided by its sum. DG-MR outputs a log-density by construction: the readout is resized, smoothed with a learned Gaussian, combined with a log-prior using a learned weight, and normalized. All models are then scored the same way, at the pixel containing each gaze sample, in bits above uniform, following [Kümmerer, 2015]: $ \mathrm{LL} = \frac{1}{\log 2}\Bigl(\frac{1}{N}\sum_{i=1}^{N} \log p(x_i) + \log (HW)\Bigr), $ with a small for numerical stability.
| SalFoM | DG-MR | |
|---|---|---|
| Fine-tuned on | SalTempto; LEDOV | SalTempto. On LEDOV we use the released LEDOV-trained checkpoint unchanged |
| Initial weights | Released DHF1K checkpoint | Released LEDOV checkpoint |
| Trainable | All weights | Readout, blur width and prior weight; Backbone frozen (as done in the original DG-MR paper) |
| Loss (kept the same as the respective papers) | KL − CC against a Gaussian-blurred gaze map | Negative LL (bits over uniform) |
At inference, both models use a sliding 16-frame window (LEDOV simply averages thier features without any temporal ordering, as done in the original paper) with stride 1 and predict its last frame, padding the first 15 frames of a video with copies of frame 0. Learning rates for DG-MR followed the published schedule, while SalFoM used 1e-5 up to convergence, as we observed model collapse with higher learning rates.
The best checkpoints were selected based on their IG over uniform on the corresponding datasets' validation splits.
Thank you for your replies
The additional details on the dataset, as well as the general reply are helpful - especially in pointing out where a read on information separating model dependence on temporal context, and information unavailable to a static predictor is found in the paper. I remain positive on this paper, and maintain my original recommendation.
Response to reviewer vGR3
We thank you for the detailed review, and for recognizing SalTempto as a solid dataset contribution: the number of videos, the high sampling rate of the gaze data, the per-observer metadata, and the long, dense recordings that support most kinds of analysis. We also appreciate your recognition of the controls, such as the repeated-frame test and the sensitivity analysis of the performance ceiling, and of the rigorous and well-motivated assessments that distinguish SalTempto from LEDOV.
We address your concerns below:
W1 - Does prediction require tracking objects?
We agree that cross-frame density alone does not support the claim that tracking is required, and we realize we misframed the point in the paper. The cross-frame transfer analysis (App.C) establishes that gaze locations change between frames, asserting dynamism in the underlying data. That is a property of the stimuli and the gaze distribution, not evidence that a static predictor cannot recover those locations from the current frame alone. The main analysis in the paper shows that handling dynamic stimuli is really needed on SalTempto, unlike in LEDOV. We have thus modified the statement in the updated paper draft.
Old statement: "...models usually do not face the challenge of tracking salient regions that drift spatially over time."
Updated statement: "...models usually can solve existing benchmarks without needing stimulus histories over time."
Q1 - Availability of temporal information as a dataset property versus use of temporal information by the model?
This is a great question, asking for two separate measurements. Both of the measurements are already provided in the paper:
The analysis in the main paper measures the second quantity: information unavailable to an independently trained static predictor. This is a property of the dataset, not of the video models being evaluated. The high performance of the static baseline on LEDOV, where it recovers 56% of the headroom above the centerbias against 13% on SalTempto (Table 2), is what shows that on LEDOV a static predictor can see much of the available information.
In the appendix, we also directly measure a given model's dependence on the temporal context of frames. We run SalFoM on a static context that repeats only the target frame (App. B, Table 8). The drop in performance shows how much the model makes use of the temporal context. On LEDOV, removing the temporal context costs SalFoM 0.26 bits, 15% of its information gain over the centerbias. On SalTempto it costs 0.48 bits, 44% of its gain, three times the share on LEDOV.
Compared with the static baseline, repeating the target frame in the full context measures a different quantity. With the target frame repeated, SalFoM is itself a static predictor, and it recovers more of the headroom than the static baseline on both datasets: 64% against 56% on LEDOV, and 33% against 13% on SalTempto. A model's dependence on temporal context therefore does not rule out information that a static predictor can see, and part of SalFoM's margin over the static baseline comes from stronger static features. Both measurements nonetheless show the same contrast between the datasets: a static predictor recovers significantly less headroom on SalTempto than on LEDOV, and temporal context accounts for a much larger share of SalFoM's gain on SalTempto (44% against 15%).
Making the difference between the properties of the dataset (measured with the static predictor) and the properties of a model (measured with the frame-repeated context) more explicit is worthwhile, and we have done so in the updated draft (\S3.2).
If we have misunderstood your question, we would welcome a clarification.