acceptodds
Withdrawn from ICLR 2027

SSC-LoRA: Semantic Spectral Control Low-Rank Adaptation for Generalizable 3D Gaze Estimation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Accurately predicting 3D gaze directions from single images remains challenging due to the absence of unified, physically grounded representations and limited diverse annotated data. Although visual foundation models (VFMs) provide strong transferable priors, their inherent visual sensitivity misaligns substantially with downstream regression tasks. Specifically, gaze-irrelevant appearance attributes dominate feature learning and overshadow subtle gaze variations that are critical for precise prediction. Large-scale pretraining prioritizes generic visual capabilities without explicitly encoding fine-grained gaze sensitivity. When appearance patterns spuriously correlate with gaze signals during pretraining, such sensitivity imbalance induces shortcut learning in downstream adaptation and degrades cross-domain generalization. To address these issues, we propose a novel parameter-efficient fine-tuning framework, Semantic Spectral Control LoRA (SSC-LoRA). Built upon vanilla LoRA, SSC-LoRA integrates gradient-based semantic identification (SI) and spectral control (SC) for task-aligned adaptation. Specifically, SI first decomposes pretrained weights into singular components and ranks their responsiveness to gaze-relevant and gaze-irrelevant objectives. SC further suppresses weight updates from gaze-irrelevant spectral bases while reinforcing gaze-specific feature learning. Both modules operate inherently within LoRA’s low-rank space without introducing additional trainable parameters. Evaluated on five gaze estimation benchmarks with CLIP-B/L and DINOv2-B/L backbones, SSC-LoRA consistently outperforms most SOTA methods in both in-domain fitting and cross-domain generalization. Extensive ablation analyses validate that the performance bottleneck stems from gaze-irrelevant-sensitive adaptation, and confirm that our spectral control mechanism effectively recalibrates update allocation across task-relevant and task-irrelevant feature subspaces.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Gaze estimation, visual foundation models, low-rank adaptation, semantic spectral control

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_wLZqOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SSC-LoRA for 3D gaze estimation. It addresses the concern that visual foundation models focus too much on appearance cues, such as identity and lighting, rather than gaze. The method uses SVD and gradients from gaze supervision and CLIP-generated appearance labels to select weight components. It then adjusts LoRA updates to suppress changes in appearance-related directions and strengthen updates in gaze-related directions, without adding trainable parameters. The authors evaluate SSC-LoRA with CLIP and DINOv2 backbones on five benchmarks, covering both in-domain and cross-domain settings, and provide ablations of the main design choices.

Critical strengths
  1. The problem is practically relevant. Adapting VFMs to gaze regression without letting them latch onto appearance shortcuts matters for cross-domain deployment.
  2. The method is simple and cheap to plug in. It reuses LoRA's A and B, adds no trainable parameters.
  3. The evaluation covers five datasets and four backbones, and the appendix includes several useful analyses, such as the PEFT comparison (Table 6), the pseudo-label accuracy check on IVGaze (Table 9), the alternative gaze supervision study (Table 8), and the layer-wise overlap analysis (Figure 8).
Critical weaknesses
  1. The motivation itself is not very new. The idea that gaze-irrelevant appearance factors (identity, illumination, accessories, expression) dominate learned features and hurt cross-domain gaze estimation is the central premise of an established line of generalizable gaze estimation work, including PureGaze, Gaze-Consistent, FSCI, and CLIP-Gaze, which the paper itself cites in the Introduction and Related Work. The paper calls this “factor-sensitivity mismatch,” but it is unclear what new insight this term adds. This makes it hard to view the observation as a separate contribution.

  2. Several claims overstate the results. The text says SSC-LoRA "also outperforms OmniGaze", but in Table 1 OmniGaze is better on MPIIFaceGaze, EyeDiap, and Gaze360, and in Table 2 it is better on both cross-domain targets by about 1°. The authors attribute this to extra training data, but this should be stated plainly rather than reported as outperformance. The IVGaze comparison with GazeDPTR in Section 4.1 refers to a method that is not in Table 1. More broadly, most baselines in Tables 1–2 use ResNet- or small Transformer backbones, so comparing them with CLIP-L/14 or DINOv2-L/14 mixes the effect of the backbone with that of the method. The fair comparison is against PEFT methods on the same backbone (Table 6), which belongs in the main paper.

  3. The gains are not consistent across backbones. In Table 6, SSC-LoRA performs well with CLIP, but its advantage is less clear with DINOv2. On DINOv2-B/14, PiSSA and DoRA achieve lower errors on Gaze360. On DINOv2-L/14, DoRA performs better in all three settings, although SSC-LoRA still improves on vanilla LoRA. In Table 1, all SSC-LoRA variants perform worse than Dilated-Net and GazeTR on GazeGene. The paper should discuss these limitations and make its claims of consistent superiority more specific.

  4. The evidence for reduced shortcut learning is still limited. The method blocks updates in selected gaze-irrelevant directions, but this alone does not show that the model relies less on appearance cues. Figure 11 is hard to interpret. Some highlighted pairs appear closer under SSC-LoRA, but the overall improvement is unclear. A clearer visualization and more direct quantitative evidence would help. For example, does SSC-LoRA make more stable predictions when lighting or other appearance factors change while gaze stays the same?

  5. The presentation needs improvement. Much of Section 3 and Appendix A (Definitions 3.2 and 3.3, A.2–A.4) formalizes standard properties of orthogonal projectors at length, while key practical details are missing from the main text. There are also minor issues: the absolute value is missing in the last term of Eq. 2, the OmniGaze citation year is inconsistent ([NeurIPS25] vs. 2026 in the references), and there are grammatical errors (e.g., "SSC-LoRA ... achieve", "Figure 4 show", "baselines includes").

Questions for authors
  1. Why do the gains largely disappear for DINOv2 (Table 6) even though Figure 10 shows similar appearance clustering? Does the use of CLIP pseudo-labels for DINOv2 scoring play a role?
  2. How does SSC-LoRA compare with LoRA in terms of total training time, peak GPU memory usage, and inference cost?
Reviewer_hjVyOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes SSC-LoRA, a parameter-efficient fine-tuning method for generalizable 3D gaze estimation. The authors identify a sensitivity mismatch in visual foundation models, where gaze-irrelevant appearance factors may dominate gaze-related features. To address this, SSC-LoRA uses gradient-based semantic identification to rank spectral components and bilateral projections to suppress gaze-irrelevant updates while enhancing gaze-relevant ones. Experiments on five gaze benchmarks with CLIP and DINOv2 backbones demonstrate improvements over vanilla LoRA, particularly in cross-domain settings.

Critical strengths
  1. Clear motivation. The paper identifies a meaningful challenge in adapting general-purpose visual foundation models to fine-grained gaze regression.

  2. Well-formulated methodology. The spectral sensitivity analysis and bilateral projection mechanism are mathematically well explained, with supporting theoretical derivations.

  3. Extensive experiments. The evaluation covers multiple datasets, backbones, and domain transfer settings, with ablations examining the proposed components.

Critical weaknesses

1. The reliability of semantic identification depends on an insufficiently validated gaze regressor.

In Section 3.1, the gaze-relevant spectral components are identified using gradients from a two-layer gaze regressor hh, trained on top of frozen CLIP/DINOv2 features. Therefore, the resulting sensitivity ranking depends heavily on how well this regressor captures gaze information.

However, the paper does not report the performance of hh before spectral scoring. This is particularly concerning given the poor linear-probing results in Table 6 (e.g., 30.40° on Gaze360 with CLIP-B/16 and 28.60° with CLIP-L/14). Although the linear-probing models are not necessarily identical to the regressor used during spectral scoring, these results raise concerns about how much gaze information can be extracted from the frozen representations.

Furthermore, Appendix B.2 states that fine-tuning starts with a newly initialized gaze regressor. This raises an additional concern about whether spectral components identified using the initial regressor remain relevant to the subsequent fine-tuning process.

Without evaluating the initial regressor and the stability of its gradient rankings, it is difficult to determine whether the selected Top-32 components meaningfully represent gaze-relevant directions.

2. Preserving gaze-irrelevant directions does not necessarily reduce the model's sensitivity to irrelevant factors.

The paper attributes poor generalization to the excessive sensitivity of pretrained visual features to gaze-irrelevant factors such as identity, glasses, and illumination.

However, Eq. 21 and Appendix A.4.2 establish that, for the selected gaze-irrelevant directions,

Wftvi=W0vi=σiui.W_{\mathrm{ft}}v_i = W_0v_i = \sigma_i u_i.

In other words, SSC-LoRA preserves the original pretrained mappings along these directions. It suppresses updates to these components rather than removing or reducing the gaze-irrelevant information already present in the pretrained representation.

This is consistent with the proposed update-control mechanism, but its connection to the claimed reduction of shortcut learning remains insufficiently demonstrated. The downstream regressor can still access these appearance-related features.

The paper does not provide direct evidence that the final gaze predictions become less sensitive to nuisance factors. For example, it could evaluate gaze prediction consistency under controlled changes in identity, glasses, or illumination, or quantify how nuisance information and gaze-relevant information are represented before and after adaptation.

3. The spectral sensitivity score may be dominated by singular-value magnitude rather than semantic relevance.

The proposed sensitivity score is

ci(L)=∣σiui⊤Gvi∣.c_i(\mathcal{L}) = \left|\sigma_i u_i^\top Gv_i\right|.

A large score may indicate a strong gradient response in a particular direction, but it may also result from a large singular value σi\sigma_i. If the singular values have a much larger dynamic range than the gradient projections, the Top-32 selection could be dominated by high-magnitude spectral components rather than meaningful semantic differences.

Figure 3(b) further shows approximately 85% overlap among the Top-32 components selected for different gaze-irrelevant factors. Although these factors may share common visual representations, such substantial overlap raises questions about the semantic specificity of the proposed identification procedure.

The paper lacks comparisons against simpler selection criteria, such as ranking by singular values alone or by the unscaled gradient projection ∣ui⊤Gvi∣|u_i^\top Gv_i|. Without these controls, it is difficult to establish whether the proposed ranking genuinely provides semantic discrimination beyond generic spectral importance.

4. The gaze-relevant enhancement mechanism uses an insufficiently justified fixed scaling factor.

Appendix A.4.2 derives

PgLΔWSSCPgR=2PgLΔWPgR.P_g^L \Delta W_{\mathrm{SSC}} P_g^R = 2P_g^L \Delta W P_g^R.

Thus, the proposed enhancement effectively doubles the LoRA update within the selected gaze-relevant block.

Although the subspace projection introduces a structural constraint, the additional enhancement is a fixed update amplification. The paper does not explain why doubling the update is appropriate or investigate alternative scaling strengths.

A scaling-factor ablation and comparisons with simpler subspace-specific gradient or update rescaling would help establish whether the observed improvements arise from the proposed semantic control mechanism or from a favorable adjustment of update magnitude.

5. Important related work on gaze-specific pretraining is missing, particularly UniGaze.

The paper does not discuss or compare against UniGaze (Qin et al., WACV 2026), which addresses a closely related problem through gaze-specific MAE pretraining on normalized facial images rather than spectral adaptation.

This omission weakens the paper's motivation: it remains unclear whether the main limitation lies in the pretrained representation itself or in how its weights are updated during fine-tuning.

6. Simple eye-focused and appearance-robust baselines are missing.

If the central problem is that pretrained features are dominated by identity, illumination, and accessories rather than eye orientation, a more direct approach would be to explicitly strengthen eye-related feature learning.

Several straightforward alternatives should be considered:

  • Eye-focused learning: Eye-crop branches, eye-region masking, or auxiliary supervision using eye landmarks or segmentation. For example, Jiang et al., Learning Unsupervised Gaze Representation via Eye Mask Driven Information Bottleneck (2024), explicitly investigates eye-centered representation learning.
  • Appearance-targeted augmentation: Training vanilla LoRA with controlled illumination changes, accessory perturbations, or gaze-consistent appearance augmentation. Related approaches, including Gaze-Consistent, have already explored reducing reliance on appearance variations.

The paper does not evaluate vanilla LoRA with these simpler modifications. Eye-specific auxiliary annotations may introduce additional supervision, so the corresponding annotation budget should be accounted for. However, appearance augmentation and self-supervised eye masking provide relatively direct alternatives without requiring the full SI+SC pipeline.

Without these comparisons, it remains unclear whether the proposed spectral control is necessary to address the identified problem, or whether comparable improvements could be obtained through simpler task-aligned training strategies.

7. The contribution of the gaze-irrelevant branch is not sufficiently demonstrated.

Table 4 shows only modest improvements from irrelevant-only suppression (0.29°/0.48° on the two cross-domain settings). Table 3 also shows relatively small differences when selecting Middle/Low components, with reported standard deviations reaching 0.37°.

More importantly, the paper lacks factor-wise ablations and a matched random-suppression baseline. Subject identity, despite being a central motivation in Figure 1, is not included among the irrelevant factors. These omissions make it difficult to establish whether the improvement comes from meaningful semantic identification or simply from restricting LoRA updates.

8. The empirical results do not consistently outperform existing methods.

SSC-LoRA does not consistently outperform existing gaze estimators. OmniGaze achieves substantially lower cross-domain errors (4.95°/5.75° vs. 5.92°/6.77° for SSC-LoRA CLIP-L), although it uses additional training data. AGG also slightly outperforms SSC-LoRA on GazeGene → MPIIFaceGaze (5.61° vs. 5.63°).

Even among PEFT methods, Table 6 shows that DoRA outperforms SSC-LoRA with DINOv2-L on all three evaluated settings, while PiSSA achieves better results in some DINOv2-B settings. These results weaken the claim of consistent superiority across backbones and benchmarks.

Questions for authors

1. How accurate and stable is the initial gaze regressor used for spectral scoring?

Please report the training protocol and MAE of the gaze regressor hh before spectral scoring, including training and validation performance. How does it differ from the linear-probing baseline in Table 6? How stable are the Top-32 spectral selections across different regressor initializations and training seeds?

2. How sensitive is the method to the strength of gaze-relevant enhancement?

Why is the enhancement coefficient fixed to produce a twofold update in the gaze-relevant block? Have the authors experimented with different enhancement strengths? Could comparable improvements be achieved through simpler subspace-specific update scaling?

3. How much of the proposed semantic ranking is explained by singular values and shared spectral components?

Could the authors report the correlation between singular-value magnitude and the proposed sensitivity score, and compare against singular-value-only and gradient-only rankings? Given the approximately 85% overlap among irrelevant factors, which factors provide distinct information for component selection?

4. Can the authors provide stronger evidence for the necessity of irrelevant-factor suppression?

Please consider factor-wise ablations, matched random-subspace suppression, and a controlled evaluation of prediction sensitivity to irrelevant appearance changes. Why is subject identity not explicitly considered, given its prominent role in the motivation?

5. How were the selection size and adapted layer ranges chosen?

Table 10 shows that k=128k=128 achieves better cross-domain performance on MPIIFaceGaze than the default k=32k=32. Similarly, Table 11 shows that adapting layers 3–12 improves cross-domain performance compared with adapting all layers.

Were these hyperparameters selected using source-domain validation data, or were target-domain results involved? Please clarify the selection protocol to rule out target-test-set leakage.

6. Can the authors report statistical variability across independent training runs?

Most main results use a single random seed and report performance from the final training epoch. Several improvements are relatively small, particularly for DINOv2. Please report mean and standard deviation across multiple training seeds, and clarify whether model selection uses validation performance or the final checkpoint.

7. What is the computational overhead of semantic identification and spectral control?

Although SSC-LoRA introduces no additional trainable adapter parameters, the method requires SVD, preliminary regressor training, pseudo-label generation, and gradient-based spectral scoring over multiple objectives. Please report preprocessing cost, fine-tuning time, and peak memory relative to vanilla LoRA.

8. Can the authors clarify the evaluation protocol relative to UniGaze?

In particular, does the EyeDiap cross-domain evaluation use the same CS subset as UniGaze, or a different test partition? An evaluation under matched preprocessing, data splits, and gaze-supervision budgets would help determine the relative benefits of gaze-specific pretraining and spectral adaptation.

Reviewer_GR4LOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The manuscript addresses 3D gaze estimation from single images under appearance and domain shift, arguing that visual foundation model features are organized by gaze-irrelevant factors and that this organization is the binding constraint on adaptation. The proposed method decomposes each pretrained weight matrix by singular value decomposition, scores every rank-1 component by its gradient sensitivity to a gaze regression loss and to supervised contrastive losses over CLIP-derived nuisance pseudo-labels, and selects top-k disjoint index sets for the two objectives. LoRA updates are then bilaterally projected so that components associated with nuisance factors are removed and components associated with gaze are doubled (Eq. 12), with the adapter parameter count unchanged. Experiments cover five benchmarks and four backbones across in-domain and four transfer settings, reporting mean angular error and ablations over selection region, control branch, selection size, adapted depth, and supervision source.

Critical strengths
  1. The formal development is correct and self-contained: the projectors in Appendix A.2 are verified orthogonal, the four-block decomposition in A.3 is exact for arbitrary index sets, and the invariance and preservation identities in A.4 follow, so the mechanism can be checked.
  2. The ablation matrix probes the mechanism from several angles rather than through a single toggle, covering selection region (Table 3), branch composition and projection side (Table 4), selection size (Table 10), adapted depth (Table 11), and supervision source (Table 8).
Critical weaknesses
  1. The core framework is “LoRA + gaze,” and LoRA-based gaze adaptation is already established. LoRA and its variants have been applied to gaze estimation. Alfa [1] explicitly adapts gaze models by reweighting semantic patterns in pre-trained filters, using SVD to extract dominant spatial components and an attention mechanism to selectively amplify components relevant to a target user. Alfa reports that it outperforms “existing TTP methods and low-rank adaptation (LoRA)-based variants” across four cross-dataset gaze benchmarks. The manuscript positions SSC-LoRA as a targeted fine-tuning framework for gaze, but the adaptation backbone is standard LoRA with low-rank factors frozen in the pretrained encoder. The novelty therefore rests entirely on the spectral control mechanism, not on the use of LoRA for gaze.
  2. The manuscript uses supervised contrastive loss (SCL) with CLIP-generated pseudo-labels to score spectral components. However, SCL has already been proposed for unconstrained adaptive gaze estimation in UAGE [2,3], which introduces “a supervised contrastive method for Unconstrained Adaptive Gaze Estimation” together with a Gaze-guided Contrastive Domain Adaptation framework. More directly, 3DPE-Gaze [4] proposes a Semantic Concept Alignment Module that “separates gaze-related and unrelated concepts through CLIP-guided contrastive learning”. This is functionally the same objective as the manuscript’s semantic identification: using contrastive signals to distinguish gaze-relevant from gaze-irrelevant factors.
  3. Every result is tested on a single training run, as Appendix B.2 states that experiments use random seed 42 and report the final epoch, while the only reported dispersion in Table 3 varies the sampled components with the training seed fixed. The margins that carry the argument are 0.09 degrees on EyeDiap against AGE-Net in Table 1 and 0.09 degrees on D360 to DD against LG-Gaze in Table 2. Reporting at least three seeds and per-fold dispersion for the 15-fold MPIIFaceGaze protocol, with a paired test for the claimed gains, is the minimum needed before the margins can be interpreted.
  4. The central diagnosis of factor-sensitivity mismatch is supported by two t-SNE panels in Figure 1 and by unsigned gradient magnitudes in Figure 4, with no quantitative measure of factor purity, no linear probe of nuisance attributes from the adapted features, and no test of whether age, gender, or illumination actually predict gaze in each dataset. Cluster geometry in a single t-SNE view is weak evidence for representational organization, since cluster sizes and inter-cluster distances are not interpretable without multiple views. The caption of Figure 1 also reports the same gaze direction within an identity cluster under different lighting, whereas Section 1 describes mixed gaze values within identity clusters, so the qualitative evidence is internally inconsistent. Reporting k-nearest-neighbour label purity for gaze and for each nuisance factor, before and after adaptation, would make contribution one testable.

[1] Hsieh, He-Yen, and H. T. Kung. "Alfa: Attentive Low-Rank Filter Adaptation for Structure-Aware Cross-Domain Personalized Gaze Estimation." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 40. No. 21. 2026. [2] Wang, Yaoming, et al. "Contrastive regression for domain adaptation on gaze estimation." 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2022. [3] Lan, Enfan, Zhengxi Hu, and Jingtai Liu. "UAGE: a supervised contrastive method for unconstrained adaptive gaze estimation." Asian Conference on Computer Vision. Singapore: Springer Nature Singapore, 2024. [4] Ge, Yangshi, Yiwei Bao, and Feng Lu. "3dpe-gaze: Unlocking the potential of 3d facial priors for generalized gaze estimation." Advances in Neural Information Processing Systems 38 (2026): 67234-67253.

Questions for authors

N/A

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.