acceptodds
Under review as a conference paper at ICLR 2027

Moving Forward with Video Saliency: A New Dataset and Benchmark where Motion Matters

Abstract

Video saliency prediction attempts to capture more natural human visual behaviour than static image saliency, and is inherently harder to model due to the additional temporal dimension. Video saliency benchmarks rest on an implicit premise that predicting gaze on video requires utilizing temporal activity distributed across frames. Prior work has challenged this premise, showing that a deliberately static baseline recovers a significant fraction of the explainable gaze information on LEDOV without integrating temporal context, and that video saliency models fail in the same places as the static baseline. We first verify that this diagnosis still stands: under a more capable gold standard than the original analysis used, and an updated panel of recent architectures, the strongest temporal architecture in the panel still does not seem to substantially improve over a fine-tuned static baseline. However, it remains unclear whether the marginal gap reflects limitations of current temporal architectures or a lack of genuinely temporal patterns in the benchmark itself. We introduce SalTempto, a new training dataset and evaluation benchmark with greater dynamism: 224 one-minute action-centric clips drawn from HACS-Segments, with gaze recordings from up to 16 subjects. On SalTempto, the same fine-tuned static baseline recovers only about 13% of the available gaze information, which is a much smaller fraction than can be recovered on LEDOV. The strongest fine-tuned temporal architecture shows a substantial gain in performance over the static baseline, indicating that it does capture meaningfully more temporal information, which LEDOV fails to measure. Yet, even this SoTA model still leaves nearly half of SalTempto’s headroom unexplained, indicating that there is still a lot of room for improvement in the field of video saliency. In addition, manual examination of SalTempto recordings allows us to describe three new human visual tendencies: object permanence despite occlusion, scene inertia, and anticipatory saccades. SalTempto releases with raw visual recordings and training and evaluation videos. It will be released as a hosted video saliency benchmark based on the provided test-set videos upon acceptance.

Primary area:
datasets and benchmarks
Keywords:
Video Saliency, Video Saliency Benchmark, Eye Movements, Gaze Model, Visual Attention

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_vGR3OpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper carries out work involving saliency modelling, specifically in the case where there is a temporal domain to the assessment. The paper suggests that there is considerable room for improvement still in establishing visual saliency when the content driving it is temporal in nature. SalTempo is introduced as a new visual benchmark suitable for this kind of assessment. This includes 224 action-centred, one minute videos including raw eye tracking, training/evaluation splits, and a benchmark of 13 model variants. Much emphasis focuses on the SalTempo model and analysis reveals that unlike the existing LEDOV evaluation, SalTempo reveals greater ability to assess saliency specific to motion information. This results in a nice dataset/benchmark contribution in allowing stronger future assessment of temporal saliency as well as a current snapshot of model behaviour.

Critical strengths
  1. This is undoubtedly a useful dataset given the number of video samples and also very high sampling rate to the gaze data itself. Some of the additional metadata that allows assessment specific to an individual observer, and the long sample length and density allows for most kinds of assessment one might want to perform.
  2. The dataset is also strong in many of the controls included such as repeated frames, and assessment of different attributes that might contribute to a ceiling on performance. Overall this is a solid dataset contribution. A meaningful distinction from LEDOV is also established through assessments that are rigorous and well motivated.
Critical weaknesses
  1. The analysis is generally good, but there are some places where the claims go a bit further than what the evidence supports. In particular, low cross-frame density establishes changing gaze locations, but doesn't necessarily guarantee that prediction requires history. Therefore, this isn't yet strong enough to state that tracking is required.
Questions for authors
  1. Can you distinguish explicitly between a model's measured dependence on the temporal context of frames and information unavailable to a trained static predictor? i.e. temporal context alone doesn't always guarantee that there is no information a static predictor can see.
  2. Can you provide additional detail on gaze-to-frame processing and eye selection, the probability normalization used for model outputs, adaptation settings, and checkpoint selection?
3 comments
AuthorsOpenReview ↗

Response to reviewer vGR3

We thank you for the detailed review, and for recognizing SalTempto as a solid dataset contribution: the number of videos, the high sampling rate of the gaze data, the per-observer metadata, and the long, dense recordings that support most kinds of analysis. We also appreciate your recognition of the controls, such as the repeated-frame test and the sensitivity analysis of the performance ceiling, and of the rigorous and well-motivated assessments that distinguish SalTempto from LEDOV.

We address your concerns below:

W1 - Does prediction require tracking objects?\textbf{W1 - Does prediction require tracking objects?}

We agree that cross-frame density alone does not support the claim that tracking is required, and we realize we misframed the point in the paper. The cross-frame transfer analysis (App.C) establishes that gaze locations change between frames, asserting dynamism in the underlying data. That is a property of the stimuli and the gaze distribution, not evidence that a static predictor cannot recover those locations from the current frame alone. The main analysis in the paper shows that handling dynamic stimuli is really needed on SalTempto, unlike in LEDOV. We have thus modified the statement in the updated paper draft.

Old statement: "...models usually do not face the challenge of tracking salient regions that drift spatially over time."

Updated statement: "...models usually can solve existing benchmarks without needing stimulus histories over time."

Q1 - Availability of temporal information as a dataset property versus use of temporal information by the model?\textbf{Q1 - Availability of temporal information as a dataset property versus use of temporal information by the model?}

This is a great question, asking for two separate measurements. Both of the measurements are already provided in the paper:

  1. A model's dependence on the temporal context of frames.
  2. Information unavailable to an independently trained static predictor.

The analysis in the main paper measures the second quantity: information unavailable to an independently trained static predictor. This is a property of the dataset, not of the video models being evaluated. The high performance of the static baseline on LEDOV, where it recovers 56% of the headroom above the centerbias against 13% on SalTempto (Table 2), is what shows that on LEDOV a static predictor can see much of the available information.

In the appendix, we also directly measure a given model's dependence on the temporal context of frames. We run SalFoM on a static context that repeats only the target frame (App. B, Table 8). The drop in performance shows how much the model makes use of the temporal context. On LEDOV, removing the temporal context costs SalFoM 0.26 bits, 15% of its information gain over the centerbias. On SalTempto it costs 0.48 bits, 44% of its gain, three times the share on LEDOV.

Compared with the static baseline, repeating the target frame in the full context measures a different quantity. With the target frame repeated, SalFoM is itself a static predictor, and it recovers more of the headroom than the static baseline on both datasets: 64% against 56% on LEDOV, and 33% against 13% on SalTempto. A model's dependence on temporal context therefore does not rule out information that a static predictor can see, and part of SalFoM's margin over the static baseline comes from stronger static features. Both measurements nonetheless show the same contrast between the datasets: a static predictor recovers significantly less headroom on SalTempto than on LEDOV, and temporal context accounts for a much larger share of SalFoM's gain on SalTempto (44% against 15%).

Making the difference between the properties of the dataset (measured with the static predictor) and the properties of a model (measured with the frame-repeated context) more explicit is worthwhile, and we have done so in the updated draft (\S3.2).

If we have misunderstood your question, we would welcome a clarification.

Q2 - Evaluation Details?\textbf{Q2 - Evaluation Details?}

We use the following evaluation pipeline:

Gaze-to-frame processing and eye selection.\textit{Gaze-to-frame processing and eye selection.} The 500Hz recordings are binocular and the released files keep both eyes. Our analyses use the left eye only, with no averaging across eyes. Missing and off-screen samples are dropped. Each sample carries a fractional video-frame index and is assigned to frame ii when i≤index<i+1i \le \text{index} < i+1, which gives about 270 samples per frame with up to 16 observers on each evaluation video. Scoring is at the native 1920×10801920\times1080. All frames with gaze are scored, including the center-anchored first frames.

Normalization to a density.\textit{Normalization to a density.} Models that output a saliency map, including SalFoM, have the map resized bilinearly to the scoring resolution and divided by its sum. DG-MR outputs a log-density by construction: the readout is resized, smoothed with a learned Gaussian, combined with a log-prior using a learned weight, and normalized. All models are then scored the same way, at the pixel containing each gaze sample, in bits above uniform, following [Kümmerer, 2015]: $ \mathrm{LL} = \frac{1}{\log 2}\Bigl(\frac{1}{N}\sum_{i=1}^{N} \log p(x_i) + \log (HW)\Bigr), $ with a small ϵ=1e−12\epsilon=1e-12 for numerical stability.

Adaptation settings:\textit{Adaptation settings:}

SalFoMDG-MR
Fine-tuned onSalTempto; LEDOVSalTempto. On LEDOV we use the released LEDOV-trained checkpoint unchanged
Initial weightsReleased DHF1K checkpointReleased LEDOV checkpoint
TrainableAll weightsReadout, blur width and prior weight; Backbone frozen (as done in the original DG-MR paper)
Loss (kept the same as the respective papers)KL − CC against a Gaussian-blurred gaze mapNegative LL (bits over uniform)

At inference, both models use a sliding 16-frame window (LEDOV simply averages thier features without any temporal ordering, as done in the original paper) with stride 1 and predict its last frame, padding the first 15 frames of a video with copies of frame 0. Learning rates for DG-MR followed the published schedule, while SalFoM used 1e-5 up to convergence, as we observed model collapse with higher learning rates.

Checkpoint selection.\textit{Checkpoint selection.} The best checkpoints were selected based on their IG over uniform on the corresponding datasets' validation splits.

Reviewer_vGR3OpenReview ↗

Thank you for your replies

The additional details on the dataset, as well as the general reply are helpful - especially in pointing out where a read on information separating model dependence on temporal context, and information unavailable to a static predictor is found in the paper. I remain positive on this paper, and maintain my original recommendation.

Reviewer_eJa5OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work tackles a very relevant problem in video saliency prediction: designing video-native benchmarks, where there is more to predict than just a central point of attention, and where the scenes temporally evolve to truly reflect the temporal nature of underlying datasets. The authors analyze their claims thoroughly -- especially around verifying that a significant fraction of the explainable gaze information on LEDOV can be predicted without integrating any temporal context.

Critical strengths

One of the most important strengths of the paper is to question the status-quo benchmarks in video saliency prediction, which, I tend to agree, are not specifically video-native. This has led to an overall stagnation of the field and I believe that by considering predictive power of static baselines, and designing a new benchmark that has a stronger temporal dynamic is the need of the hour. The authors make a meaningful step towards this goal.

Critical weaknesses

While LEDOV is a landmark benchmark in the field, I believe that DIEM, DHF1K are also crucial and I would request the authors to augment their analysis with this benchmarks also (at least DIEM). Apart from this, a critical weakness seems to be dataset size and clip duration: while I appreciate the principled approach the authors have taken to designing a video-specific benchmark, the size of it is working against my assessment of the contribution.

Questions for authors
  1. Suggestion: please add some visual samples of the dataset in the paper
  2. The paper is overly fixated on LEDOV as a baseline dataset -- please consider diversifying with DIEM at least.
  3. How many observers are sufficient in a saliency map is dependent on scene complexity, inter-observer agreement, etc. [1] Please discuss how did the authors arrive at the choice of number of observers.
  4. Please elaborate on plans of expansion since the current dataset size and clip duration somewhat constrained.
  5. Sometimes temporal saliency can be predicted by detecting and tracking the prominent object in a scene (for certain sports videos as well). Please elaborate on how the scene content was curated to avoid this effect.

[1] https://arxiv.org/pdf/2104.08038

3 comments
AuthorsOpenReview ↗

Response to reviewer eJa5

We thank you for endorsing the core motivation that status-quo video-saliency benchmarks are not video-native and that the field needs benchmarks with stronger temporal dynamics, and for viewing this work as a meaningful step towards this goal. We also appreciate your recognition of the thorough analysis of our claims, in particular the verification that a significant fraction of LEDOV's explainable gaze information can be predicted without temporal context, and of the principled approach we took to designing a video-specific benchmark. Several of your concerns are addressed by material already in the submission, which we have made more prominent in the updated draft. We address your concerns below:

W1,Q2 - Why only LEDOV?\textbf{W1,Q2 - Why only LEDOV?}

The use of only LEDOV is not an arbitrary choice: among existing benchmarks we are aware of, it is the only one on which this analysis can be run without the confounders described in $\S$2. SalTempto is thus a necessary contribution, as no other set besides LEDOV currently supports such an analysis without these confounders. SalTempto now provides an additional benchmark video saliency models can be ranked more accurately on.

DIEM was already in the submitted paper: $\S$2 discusses it, and App.~B (paragraph ``DIEM'') reports a zero-shot comparison on its test videos. It was easy to miss there, so we have moved it into the main text, in the results section ($\S$4) of the updated draft.

Share of headroom recovered (IGE)LEDOVDIEMSalTempto
DG-MR56%22%13%
SalFoM75%54%58%
Gap in percentage points193245

LEDOV remains the main comparison for the reasons in $\S$2: the static-recoverability diagnostic was established on it, it is a free-viewing benchmark with per-observer gaze, and it is the usual fine-tuning target for new architectures.

We treat DIEM as supporting evidence, not as a third primary benchmark. DIEM was recorded with audio playback and heterogeneous equipment and recording setups, and its model ranking correlates only weakly with the other two datasets (Spearman ρ=0.44\rho = 0.44 with LEDOV and 0.620.62 with SalTempto).

As noted in $\S$2, the released fixation data in DHF1K has known artefacts and does not keep per-subject identity, which our leave-one-subject-out gold standard requires. Ten of the thirteen models we benchmark were also trained primarily on it ($\S$3.2), so it would not be a zero-shot test benchmark. This is why we prioritized LEDOV, and to a lesser extent, DIEM.

W2,Q4 - Is the dataset too small? Expansion plans?\textbf{W2,Q4 - Is the dataset too small? Expansion plans?}

Having fewer clips is a deliberate choice. Under a constant recording budget ($\S$3.1), we concentrated our time and resources on longer clips, with more varied content within each clip, rather than on more clips of shorter length. For motion to matter, there has to be enough time for interesting events to occur and for semantics to unfold. Each SalTempto clip lasts one minute and covers an action's lead-up and aftermath ($\S$3.1), whereas LEDOV's median clip lasts about 10s (mean about 12s), which leaves little room for this.

SalTempto therefore holds more video than LEDOV despite having fewer clips: 3.4h against 1.52h for training, and 0.17h against 0.13h per evaluation split ($\S$3.1). It also holds more video in total than DIEM, about 3.7h against 2.2h.

Statistically, we find that the sizes of the evaluation splits in SalTempto are sufficient to establish the paper's central point, that a temporal model's lead over the static baseline is far larger on SalTempto than on LEDOV, since the jackknife intervals in Table 2 do not overlap. This shows that the inferences drawn based on SalTempto cannot be attributed to noise or random variation in the videos used for the analysis.

SalTempto is meant as a hosted benchmark ($\S$3.1), and we plan yearly releases as a video benchmark challenge. This paper is the first step towards that, and serves as a grounding and a current snapshot of the field. More recording is already underway: we plan to roughly double the data size and release the next version in the second half of 2027. Because this depends on a fixed recording setup and homogeneous equipment, it is slower and more time-consuming to record than data collected across heterogeneous setups, such as DIEM ($\S$2).

Q1 - More visual samples.\textbf{Q1 - More visual samples.}

SalTempto frames with human gaze or its gold-standard density overlaid are already shown in Figs. 3-6, alongside model predictions. The supplementary material submitted in the zip includes three validation videos overlaid with gaze and gold-standard densities. As the figures read primarily as results, we have added an overview figure to the updated draft (Fig. 7, App. C), showing 8-frame sequences with gaze overlays. We thought that the existing samples already cover the dataset well. Did you have something specific in mind?

Q3 - How many observers suffice?\textbf{Q3 - How many observers suffice?}

We appreciate the pointer to [1]. We can check this directly. To test whether 16 observers are enough, we build the gold standard from increasing numbers of observers and check when performance stops improving. We hold out two randomly chosen observers and build the gold standard ($\S$3.2) from 1, 2, 4, 8, 12 or 14 of the remaining 14 observers, scoring each version on the two held-out observers. As a final point, we score the same two observers with the paper's leave-one-subject-out gold standard, which uses the other 15 observers. On the validation split, the LL gained per added observer shrinks from 0.039 bits between 4 and 8 observers to 0.013 bits between 14 and 15, and on both splits LL with 12 observers is within 0.04 bits of its value with 14. The evaluation splits therefore sit on the flat part of this curve, with diminishing returns as observers are added. This flattening supports 16 observers as sufficient for the evaluation splits.

[1] also warns that directly fitting noisy maps overfits to the noise. That concern is specific to objectives trained against an estimated density, such as KL. Log-likelihood maximization (Kümmerer et al., 2016) needs no density estimate: it scores the predicted density at each fixation actually made. An analogy can be made to language models that use cross-entropy on the token that occurred, not a KL over all possible tokens. This paper, and the additional results on SalFoM provided in the rebuttal, thus also suggest that the field should move towards better training objectives than relying on noisy estimates of true fixation density.

We also tested how much the number of training observers matters across both objectives (see also, our response to DYpg-W1). Our new experiment on fine-tuning SalFoM with 1, 2 and up to 3 observers per video gives validation LL of 2.393, 2.441, and 2.443 bits with the log-likelihood objective, and 2.359, 2.329, and 2.358 bits with KL −- CC (the original objective of SalFoM). At these counts, KL −- CC loses nothing with one observer, and log-likelihood gains little beyond two. The large number of training videos provides enough visual diversity that a few observers per video are enough to learn good models of the behaviour from the consistency across\textbf{across} the videos. Each fine-tuned model in the paper results keeps its native objective (SalFoM: KL −- CC; DG-MR: LL), so the main results reflect the pipelines as released.

Training observers per video0 (no fine-tuning)123
Log-likelihood objective2.2782.3932.4412.443
KL − CC objective (original)2.2782.3592.3292.358

Q5 - Is gaze just object tracking?\textbf{Q5 - Is gaze just object tracking?}

If seeing the prominent object were enough, the static baseline could predict gaze: it would not even need to track the object, as the object is still prominent in the next frame. This appears to be what happens on LEDOV, where the static baseline recovers 56% of the headroom above the centerbias. On SalTempto it recovers only 13% (Table 2), so seeing the prominent object is not enough here, even though we did not explicitly curate scenes to avoid this effect. The only curation was the position of the annotated action segment, so that each clip contains the event together with its lead-up and aftermath ($\S$3.1).

We now perform additional analysis for the notion that gaze is only moderately concentrated on objects. If gaze were "detect-and-track the prominent object'', it would be heavily concentrated on objects. We perform a new experiment which shows it is almost uniform relative to object area. We ran a class-agnostic tracker (SAM 2.1; Ravi et al., 2024) without prompts on the 20 evaluation videos and measured where gaze falls. 62.5% of gaze samples land inside a tracked object, against 53.7% if gaze were uniform over the frame (i.e., random chance). Following the tracker's objects accounts for less than two-thirds of the gaze samples, and 37.5% of the gaze samples fall outside all tracked objects.

The paper's evidence against gaze being dictated by detecting and tracking the prominent object is the three behaviours in $\S$4, where following the visible prominent object would not be enough: gaze stays on an occluded target, stays at pre-cut positions after a scene change, or leads a moving shuttle while models concentrate on the players. Additionally, our data shows that gaze is highly multimodal, with multiple possible locations where an observer can look. There is no single "prominent object" that is consistently tracked across a scene by observers. This can be seen in the multimodal density estimates of our gold standard estimate in the paper.

We thank you again for the constructive questions.

As the discussion period closes tomorrow, we would like to kindly follow up on our rebuttal. We have carefully addressed each of your concerns through detailed responses clarifications, and additional experiments. We have also pointed out places where results for requested experiments already exist in the submitted draft.

We would greatly appreciate it if you could review our responses and let us know if you have any remaining concerns or clarifications.

Reviewer_DYpgOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper provides a new video saliency dataset which provides a large range of motion and context for short videos. This dataset is analyzed in terms of efficiency to train models of saliency.

Critical strengths

The proposed dataset is well constructed.

Critical weaknesses

While the dataset is well constructed the work is not finished ... The training data only has 3 viewers and tests still remain to be done.

Questions for authors

089 : not sure that those databases are so "recent" ...

In Figure 1 Salfom appears with no reference, we need to way a lot of occurences of salfom before finding one.

The comparison of saltempo and ledov makes sense but :

  • there is no comparison with the other cited datasets which are used in the video saliency benchmark.
  • While IG metric makes sense it is 1) used without any reference and 2) other classical saliency metrics should have been used as for the other tests.

149 : fixation collection not clear : why it is already split in train/validation ? Here the authors should state already that the purpose is a benchmark for ML-based models. Also, even if I understand why the training has a reduced number of viewers, 3 is not enough as at least 10 viewers is the standard for human gaze results to be representative.

It is good to allow breaks as 50 videos and the centered crosses is too long for one day test. The experiment is well built and the hardware is very precise.

What is table 5 in the annex ? There is not Table 5 caption ...

3 comments
AuthorsOpenReview ↗

Response to reviewer DYpg

We thank you for the careful read, and for recognizing that the dataset is well constructed, that the experiment is well built, with precise hardware and breaks for participants across the recording sessions, and that both the comparison between SalTempto and LEDOV and the use of the IG metric make sense. We address your concerns below:

W1 - Are 3 training viewers enough?\textbf{W1 - Are 3 training viewers enough?}

3 subjects per video is sufficient for training as shown by our new results.\textbf{3 subjects per video is sufficient for training as shown by our new results}.
16 subjects for validation/tests is sufficient for density estimation.\textbf{16 subjects for validation/tests is sufficient for density estimation.} The convention of at least 10 viewers exists to make empirical density estimates reliable, which is why we meet it on validation and test.

Under a constant recording budget, we favored more training videos over more viewers per video ($\S$3.1), as has been established in the literature (Agrawal et al., 2026; Schwetlick et al., 2024). Whether training needs more viewers depends on the objective. There is a tradeoff between fewer observers over more diverse video, versus many observers over few videos. Training splits benefit from a broader range of visual content, as training does not require true density estimates. Log-likelihood maximization (Kümmerer et al., 2016) needs no density estimate during training: it scores the model's predicted density at each fixation actually made. The analogy is language model training, which uses cross-entropy on the token that actually occurred, not a KL over all possible tokens. Here too, log-likelihood needs the fixations that were made, not a distribution over all possible fixation coordinates.

Training objective matters more than the number of observers in training data.\textbf{Training objective matters more than the number of observers in training data.} To test the impact of the number of observers on training data directly, we fine-tuned SalFoM under each objective (the DG-MR LL, and its original KL-CC) with 1, 2, and 3 viewers per training video. Validation LL is 2.393, 2.441 and 2.443 bits with log-likelihood, and 2.359, 2.329 and 2.358 bits with KL −- CC, against 2.278 bits with no fine-tuning (Table 3). One viewer per video is already enough: KL −- CC reaches the same LL as with all training viewers, log-likelihood does better still, and it gains little beyond two viewers (please refer to eJaY-Q3 for a tabular view.). Each fine-tuned model keeps its native objective (SalFoM: KL −- CC; DG-MR: LL), so the main results reflect the pipelines as released.

Q1 - Are the datasets recent?\textbf{Q1 - Are the datasets recent?}

We cite the most recent\textbf{most recent} datasets in the field that we are aware of. However, we realize that the term "recent" might be subjective and a few years may not be "recent" anymore in the modern ML landscape. We have therefore removed the "recent" term from the paper.

Q2 - SalFoM citations?\textbf{Q2 - SalFoM citations?}

We have added SalFoM citations (Moradi et al., 2024) at its first mention, in the Fig. 1 caption.

Q3 - Other datasets besides LEDOV?\textbf{Q3 - Other datasets besides LEDOV?}

The comparison with DIEM was already in the submitted paper (App. B): zero-shot under the same pipeline, DG-MR recovers $\approx$22% and SalFoM $\approx$54% of the gold-standard headroom. DG-MR's recovery and its gap to SalFoM fall between those on LEDOV and SalTempto (Table 2), which is consistent with the dynamism account, on a dataset we did not design. The LEDOV and SalTempto values are fine-tuned, while DIEM's are zero-shot. We now update the placement: the updated draft moves these results from the appendix into the main text ($\S$4). However, it should be noted that DIEM has two confounders, as mentioned in our Section 2: (1) it contains audio stimuli in addition to video, and is thus not purely gaze, and (2) it uses different equipment and recording setups across videos.

Hollywood-2 and UCF-Sports are excluded for the reasons in $\S$2: task-driven instead of free-viewing gaze, and single-domain content. DHF1K's released fixation data has known and substantial artefacts and lacks per-subject identity, which precludes the leave-one-subject-out gold standard our analysis relies on.

We further refer you to Q2 of reviewer eJa5 for the reason why LEDOV is the only primary comparison in the paper.

Q4 - IG reference and classical metrics?\textbf{Q4 - IG reference and classical metrics?}

Both are already in the paper. IG and IGE are defined and attributed to the information-theoretic framework of Kümmerer et al. (2015) in $\S$3.2 and App. A (Eq.1). CC, NSS, AUC, SIM and KLD are reported for every model on every (dataset, split) pair in Tables 3-6. We now cite IG in its first appearance in the updated draft.

Q5 - Why a fixed split?\textbf{Q5 - Why a fixed split?}

Datasets with a fixed train/validation split are extremely common in the machine learning community, because a fixed split makes models directly comparable. A split is also useful beyond just ML-based models: mechanistic and cognitive models of gaze can also have hyperparameters that need tuning. A fixed split also lets us design each split for its own purpose. Training benefits from many videos with a few observers each, while evaluation needs many observers per video for reliable density estimates (W1), so SalTempto has up to 3 observers per training video and up to 16 per validation and test video. That SalTempto is a benchmark for video saliency models is stated in the abstract and $\S$1.

Q6 - Table 5’s caption?\textbf{Q6 - Table 5's caption?}

Table 5 (LEDOV validation results) has its caption above the table, as do all nine tables, and every in-text table reference resolves to the intended table.

W1 - More work needed?\textbf{W1 - More work needed?}

The requested items in your questions are already in the paper.\textbf{The requested items in your questions are already in the paper.}

  1. Comparisons with other datasets: the zero-shot DIEM results were in the appendix and are now in the main text, and DHF1K, Hollywood-2 and UCF-Sports are not suitable for the reasons in $\S$2 (Q3);
  2. A reference for IG, and the classical saliency metrics: IG is attributed to Kümmerer et al. (2015) (now at its first occurance), and CC, NSS, AUC, SIM and KLD are reported for every model on every (dataset, split) pair in Tables 3-6 (Q4).

The submission also reports a leave-one-video-out uncertainty analysis (Table 2), the sensitivity of the gold standard to its deep component (Table 7), a frame-repeated temporal-context test (Table 8) and the dataset-level dynamism analysis (App. C).

We could not identify other outstanding tests from the review. Did you have something else in mind? If so, please tell us which tests or evaluations you would like to see, and which parts of the paper you consider unfinished.

As the discussion period closes tomorrow, we would like to kindly follow up on our rebuttal. We have carefully addressed each of your concerns through detailed responses, clarifications, and additional experiments. We have also pointed out places where results for the requested experiments already exist in the submitted draft.

We would greatly appreciate it if you could review our responses and let us know if you have any remaining concerns or clarifications.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.