acceptodds
Under review as a conference paper at ICLR 2027

Learning Visual Relations with Adaptive Semantic Calibration for Few-Shot Action Recognition

Abstract

Few-shot action recognition (FSAR) aims to recognize novel action categories from only a few labeled videos. Recent approaches have benefited from set-based video matching and semantic knowledge derived from pretrained vision-language models (VLMs). However, semantic knowledge remains unevenly utilized in FSAR: it is underused in support-query relation learning, insufficiently adapted to video-level recognition, and combined with visual evidence under globally shared supervision that ignores query-specific differences. To address these limitations, we propose Semantic-Adaptive Relation Fusion (SARF), which couples semantic-guided visual relation learning with semantic response recalibration and instance-adaptive semantic–visual supervision. Specifically, the Text-Guided Feature Alignment (TFA) module incorporates textual knowledge into support-query relation modeling to strengthen discriminative local correspondences between videos. The Semantic Logits Recalibration (SLR) module learns residual corrections for VLM-derived semantic logits, adapting semantic responses to the target FSAR task and alleviating confusion among semantically related categories. Finally, the Semantic–Visual Fusion (SVF) module estimates instance-dependent weights from visual relation representations and constructs adaptive semantic–visual predictions that provide query-specific auxiliary supervision during training. Extensive experiments on multiple FSAR benchmarks demonstrate that SARF achieves state-of-the-art performance across most evaluated settings, validating the effectiveness of jointly learning fine-grained visual relations and adaptively coordinating semantic and visual evidence under few-shot supervision.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Few-Shot Learning, Action Recognition, Semantic Calibration

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_usS6OpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a Semantic-Adaptive Relation Fusion (SARF) method, which couples semantic-guided visual relation learning with semantic response recalibration and instance-adaptive semantic–visual supervision. The framework consists of three modules: TFA, SLR, and SVF. Extensive experiments demonstrate the effectiveness of the proposed method, and detailed experimental settings ensure its reproducibility.

Critical strengths
  1. Comprehensive experiments. The paper evaluates on multiple benchmarks (SSv2, HMDB51, Kinetics-400) under both standard and generalized few-shot settings, with thorough ablation studies (Table 2) that isolate the contribution of each proposed module (TFA, SLR, SVF). This provides convincing evidence that the performance gains are not due to a single component but stem from the synergistic design.

  2. Clear and coherent writing. The paper is well-organized and easy to follow. The motivation is clearly stated, the method description flows logically from high-level intuition to technical details, and the figures/tables effectively illustrate the key ideas. This makes the work accessible even to readers not deeply familiar with the FSAR literature.

Critical weaknesses
  1. The technical contribution appears incremental relative to TS-FSAR and existing semantic adaptation techniques. SARF retains TS-FSAR’s core feature extraction and episodic matching framework, together with its base training objectives. Its main additions combine attention-weighted text conditioning for relation features, residual logit adaptation, and instance-conditioned semantic–visual fusion for auxiliary supervision. Although these choices are reasonable, the paper does not clearly establish what technical insight their integration provides beyond adapting familiar mechanisms to this framework. Table 2 includes useful comparisons with alternative designs, but the ablations are restricted to SSv2-Small 5-way 1-shot, and the additional gain from SVF is 0.44 percentage points over TFA+SLR. Results across multiple training seeds and additional settings, together with a quantification of the added training cost, would help establish the robustness and practical value of the proposed combination.

  2. The motivation and positioning relative to prior semantic-aware FSAR methods need to be sharpened. The broad rationale of using pretrained vision–language semantics to compensate for limited support examples is already well established, as exemplified by CLIP-FSAR and TS-FSAR. The paper acknowledges these advances and instead emphasizes the limited direct involvement of textual semantics in relation construction. However, it does not sufficiently establish why existing forms of semantic conditioning are inadequate for this purpose, or what specific limitation TFA resolves beyond the TS-FSAR framework on which it is built. A more precise comparison of where and how semantics enter these methods, together with a clearer explanation of the resulting limitations, would strengthen the claimed research gap.

Questions for authors
  1. The paper positions itself against CLIP-FSAR, MAFSAR, and TS-FSAR, but the technical distinctions remain somewhat vague. Could the authors provide a more concrete comparison of algorithms to clarify what is genuinely new?

  2. Table 4 in the Supplementary Materials ablates the effect of different text descriptions. Could the authors report a variant where the detailed descriptions are replaced by simpler ones (e.g., "a video of [action]" or just "[action]")? This would help quantify how much the carefully crafted prompts contribute to the final performance in this paper.

2 comments
AuthorsOpenReview ↗

Response to Reviewer usS6

Thank you for the careful assessment. We have sharpened the contribution statement, added robustness and cost measurements, and completed the requested prompt control.

The individual operations are intentionally lightweight. The contribution is their coordinated use to address three limitations of existing semantic-aware FSAR: semantic information does not directly participate in local relation construction, transferred semantic responses are not adapted to the video task, and semantic-visual auxiliary supervision lacks a query-dependent balance. TFA addresses the first limitation by selecting category descriptions using video-conditioned attention and injecting them before relation construction. The variants without semantic attention, with direct concatenation, and with additive fusion obtain 59.85%, 59.74%, and 59.81%, respectively, compared with 60.89% for TFA. The learned global scalar reaches 61.32%, whereas query-dependent SVF reaches 61.99%, further supporting instance-dependent supervision.

To evaluate robustness across independent training runs, we repeated both TS-FSAR and SARF using training seeds 0, 1, and 2 on SSv2-Small and HMDB-51. For each dataset, all checkpoints were evaluated on the same 10,000 test episodes:

DatasetMethodSeed 0Seed 1Seed 2Mean ± std.
SSv2-SmallTS-FSAR59.9960.3159.8460.05 ± 0.24
SSv2-SmallSARF61.9961.8061.0361.61 ± 0.51
HMDB-51TS-FSAR85.0484.5785.0384.88 ± 0.27
HMDB-51SARF85.6785.7885.4385.63 ± 0.18

SARF improves over TS-FSAR in every paired run. The average gains are 1.56 ± 0.41 percentage points on SSv2-Small and 0.75 ± 0.42 percentage points on HMDB-51, demonstrating that the improvement is not confined to a particular training seed or dataset.

We additionally report both training and inference costs under identical hardware and settings:

MethodParams (M)GFLOPsInference (ms/episode)Train (ms/iter.)Train memory (GB)
TS-FSAR101.551569.55168.25 ± 0.44382.812.77
SARF110.141578.63170.46 ± 2.84434.813.97

SARF adds 1.3% inference latency. Training time and peak training memory increase by 13.6% and 9.4%, respectively. These measurements are now included in the supplementary material.

To directly examine prompt sensitivity, we evaluated SARF using the two simplified prompt forms requested by the reviewer. The raw action name [action] and the template a video of [action] achieve 60.75% and 61.35%, respectively, compared with 61.99% using the inherited detailed descriptions. Although richer descriptions provide additional semantic information, simplifying them results in relatively small reductions of 1.24 and 0.64 percentage points. This indicates that SARF remains effective without relying exclusively on detailed descriptions. Importantly, TS-FSAR and SARF use exactly the same inherited descriptions, frozen text encoder, and aggregation procedure in the main comparison. Therefore, the improvement of SARF cannot be attributed to access to different or richer textual information.

Prompt formAccuracy (%)
Raw action name: [action]60.75
Simple template: a video of [action]61.35
Inherited detailed descriptions61.99
AuthorsOpenReview ↗

Rebuttal follow-up

Dear Reviewer,

I hope this message finds you well.

As the discussion period is drawing to a close, we would like to respectfully check whether our rebuttal has helped address the initial concerns you raised. Please let us know if you have any further questions or if there is anything else we can clarify. We would be more than happy to discuss further. Your perspective is crucial for us to improve the paper.

Thank you again for your time and valuable review!

Best regards!

Reviewer_XJSrOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper proposes Semantic-Adaptive Relation Fusion (SARF), built on TS-FSAR, for few-shot action recognition. Text-Guided Feature Alignment (TFA) introduces weighted class-description embeddings into relation representations and applies a consistency loss. Semantic Logits Recalibration (SLR) uses two residual logit adapters, while Semantic–Visual Fusion (SVF) supplies instance-dependent auxiliary supervision during training. Inference instead combines relation, CLIP, and LSN predictions through a fixed weighted product. Experiments cover five benchmark configurations under 5-way 1-shot and 5-shot protocols. The paper reports improvements over its reproduced TS-FSAR baseline in all ten settings, supported by component ablations and additional analyses.

Critical strengths
  1. TFA brings category-level textual semantics into relation construction during training by weighting projected class-description tokens according to visual relevance and aggregating them with local visual tokens (Section 3.2, Equations (1)–(2)). This is a well-motivated way to make local relation representations sensitive to action meaning when support examples are scarce.

  2. SARF improves over the reproduced TS-FSAR baseline in all ten evaluated settings in Table 1, including 2.0 and 1.6 percentage points on SSv2-Small and SSv2-Full 1-shot. Table 2(b) also supports the content-dependent semantic weighting: TFA reaches 60.89%, compared with 59.85% without semantic attention. Appendix B fixes the textual descriptions and their aggregation across SARF variants. These cross-benchmark point-estimate gains and focused design comparisons provide empirical support for the practical usefulness of the semantic integration described above.

Critical weaknesses
  1. The paper does not clearly explain how training SLR improves the model used at inference. Figure 2 and Section 3.5 show that inference combines SrelS_{\mathrm{rel}}, SclipS_{\mathrm{clip}}, and SlsnS_{\mathrm{lsn}}, without directly using the recalibrated logits SslrS_{\mathrm{slr}}. Yet the SLR-only variant improves accuracy from 59.99% to 60.64% in Table 2(a). Appendix B retains KL consistency, so an indirect training connection is possible. However, the manuscript does not state whether that connection uses the SLR output or which trainable parameters are shared with the inference branches. The authors need to identify which inference-time parameters are updated by Lslr\mathcal{L}_{\mathrm{slr}} and through which path. This clarification is necessary to interpret the reported gain and reproduce the component’s contribution.

  2. The methodological contribution of TFA and SLR appears incremental relative to the cited prior work. CLIP-FSAR already uses text to refine visual prototypes, and Logits DeConfusion with CLIP for Few-Shot Learning already applies residual corrections to CLIP logits. TFA instead incorporates content-weighted text tokens into local relation construction, while SLR uses two successive residual logit adapters. These are useful design choices, but the manuscript does not clearly explain what new mechanism or insight they establish beyond adapting and combining existing ideas for this task. Tables 2(b) and 2(c) already compare alternative TFA designs and a DeConfusion-style variant, providing evidence for the effectiveness of these choices. The case for a substantial methodological advance is less convincing, which limits my assessment of the contribution despite the positive results.

Questions for authors
  1. In the SLR-only row of Table 2(a), which parameters used by the inference model are updated by gradients from Lslr\mathcal{L}_{\mathrm{slr}}? Appendix B retains KL consistency: please specify its actual inputs, any trainable CLIP adaptation or shared parameters, and detach choices that connect SLR to the prediction path. This clarification is essential to interpret the 59.99% to 60.64% improvement.

  2. What is the principal methodological advance of TFA’s relation-level semantic integration over text-guided prototype refinement in CLIP-FSAR, and of SLR’s two-stage residual adaptation over the correction mechanism in Logits DeConfusion with CLIP for Few-Shot Learning? Tables 2(b) and 2(c) already provide useful design comparisons. Please explain what additional mechanism or insight the proposed adaptations contribute, beyond the observed accuracy gains.

  3. Appendix B gives the chosen dual-α\alpha settings, and Figure 6 studies their sensitivity. Do the Table 2 baseline and component variants use the same dual-α\alpha configuration and inference fusion weights? This would help interpret the component comparisons. As supplementary evidence, please report any available results on variation across independent training runs for these comparisons, given that Appendix B specifies seed 0.

  4. Section 4.4 and Figure 4 explicitly define ranking over all training categories. Which query split is used for Figures 4 and 7, and how should this be reconciled with Appendix A's statement that all reported results use disjoint meta-test classes? If these are seen-class diagnostics, please label their scope. If the queries instead come from meta-test classes, please explain the candidate label space and how the true query class can be ranked within it.

4 comments
AuthorsOpenReview ↗

Response to Reviewer XJSr

Thank you for the detailed technical questions. We have revised the method description to make the optimization paths, methodological distinctions, and diagnostic protocol unambiguous.

SLR receives SclipS_{clip} without detachment. Consequently, LslrL_{slr} backpropagates through SclipS_{clip} and updates the learnable visual adapter retained at inference, while the CLIP encoder remains frozen. The two SLR adapters themselves are training-only. The inherited KL term aligns the training-category visual classification logits with the raw CLIP semantic logits SclipS_{clip}; SslrS_{slr} does not enter this term. Instead, SslrS_{slr} is optimized by its semantic classification objective and also contributes to the auxiliary SVF prediction. The only stop-gradient operation is used in the TFA consistency objective,

Ltfa=MSE(Norm(Ftfa),sg[Norm(Fv)]),L_{tfa}=MSE(Norm(F_{tfa}),sg[Norm(F_v)]),

where FvF_v serves as the visual target. These implementation details are now stated explicitly in Sections 3.1, 3.3, and 3.5 and Eq. (8).

Our distinction from prior work concerns where semantic information operates in episodic video recognition, rather than claiming attention or residual adapters as new primitives. CLIP-FSAR refines class-level prototypes after visual feature construction. In contrast, TFA selects category descriptions with video-conditioned attention and injects the weighted text tokens before the α-distance relation operator, allowing semantics to participate directly in local relation construction. Logits DeConfusion uses corrected CLIP logits primarily as prediction outputs; SLR instead acts as a training-time auxiliary head that optimizes the retained semantic pathway and supplies recalibrated supervision to SVF. SARF therefore coordinates semantics at relation construction, response adaptation, and query-dependent supervision.

All Table 2 comparisons use the same data split, descriptions, training schedule, and final inference weights (ηrel,ηclip,ηlsn)=(0.4,0.2,0.4)(η_{rel},η_{clip},η_{lsn})=(0.4,0.2,0.4). The reproduced TS-FSAR baseline and variants without TFA retain the original single-α relation formulation. TFA and dual-α relation enhancement are enabled together, using the same (αbase,αextra,β)=(0.4,1.0,0.05)(α_{base},α_{extra},β)=(0.4,1.0,0.05) configuration in all corresponding variants, without row-specific tuning. This setting is now stated in Table 2 and its caption.

The individual component variants in Table 2 are controlled seed-0 runs under the identical configuration described above; they were not independently retrained for every seed. As the available analysis of variation across independent training runs, we trained the complete TS-FSAR and SARF models from scratch using seeds 0, 1, and 2 on SSv2-Small and HMDB-51. For each dataset, all checkpoints were evaluated on the same 10,000 test episodes generated using a fixed evaluation seed:

DatasetMethodSeed 0Seed 1Seed 2Mean ± std.
SSv2-SmallTS-FSAR59.9960.3159.8460.05 ± 0.24
SSv2-SmallSARF61.9961.8061.0361.61 ± 0.51
HMDB-51TS-FSAR85.0484.5785.0384.88 ± 0.27
HMDB-51SARF85.6785.7885.4385.63 ± 0.18

SARF outperforms TS-FSAR in every paired run, with average gains of 1.56 ± 0.41 percentage points on SSv2-Small and 0.75 ± 0.42 percentage points on HMDB-51. These results provide independent-run evidence for the stability of the complete-method comparison across both training seeds and datasets.

Figures 4 and 7 provide descriptive mechanism diagnostics of SLR rather than additional benchmark evaluations. During training, the SLR adapters are defined over the dataset-specific meta-training categories and are removed for novel-class inference. We therefore analyze their direct before/after behavior in the same meta-training-category semantic space, using 1,000 diagnostic episodes and 5,000 query predictions sampled from the meta-training split. These samples are used only to inspect how SLR changes semantic response ordering and are not treated as held-out evidence of generalization. All benchmark accuracies in Tables 1 and 2 are computed separately on the disjoint meta-testing categories. At meta-test time, the SLR adapters are removed, while the visual adapter optimized through SLR supervision is retained; the resulting SclipS_{clip} is computed over the candidate novel classes and participates in the final prediction. We have clarified this scope in Section 4.4.

AuthorsOpenReview ↗

Rebuttal follow-up

Dear Reviewer,

I hope this message finds you well.

As the discussion period is drawing to a close, we would like to respectfully check whether our rebuttal has helped address the initial concerns you raised. Please let us know if you have any further questions or if there is anything else we can clarify. We would be more than happy to discuss further. Your perspective is crucial for us to improve the paper.

Thank you again for your time and valuable review!

Best regards!

Reviewer_XJSrOpenReview ↗

Follow-up on the Author Response

Thanks for the clarifications and the additional experiments. The rebuttal explains how SLR affects inference and clarifies the data split used in Figures 4 and 7. The multi-seed results support the gains of the full SARF model on both datasets. TFA moves text guidance from class-level prototype refinement to local relation construction, and SLR uses residual logit correction as auxiliary supervision during training. These are useful adaptations for few-shot action recognition, but I remain unconvinced that either constitutes a substantial methodological advance over CLIP-FSAR or Logits DeConfusion, respectively. Table 2 also enables TFA and the dual-α formulation together, so it is difficult to isolate TFA’s contribution. I am keeping my original rating of 2 (Weak rejection).

AuthorsOpenReview ↗

Follow-up response to Reviewer XJSr

Thank you for this follow-up. We apologize that we misunderstood the reviewer’s concern in our previous response and therefore did not report an existing control experiment that directly separates the contributions of TFA and dual-α. This experiment had already been conducted as part of our original ablation study, and we provide the result below.

We respectfully maintain that the methodological contribution goes beyond the reuse of attention or residual adapters. The key difference lies in the information flow and optimization role of semantics in episodic FSAR. CLIP-FSAR introduces textual knowledge at the class-prototype level after visual relation features have been formed. In contrast, TFA uses each video to select and weight category text tokens and injects them before the relation operator, allowing semantics to participate directly in constructing local support-query relation representations. Similarly, unlike Logits DeConfusion, which primarily uses corrected logits for prediction, SLR converts logit recalibration into training-time supervision that optimizes the semantic pathway retained at inference and provides task-adapted semantic responses to SVF. SVF further makes this semantic-visual supervision query-dependent. SARF therefore coordinates semantics at three distinct stages: relation construction, response adaptation, and cross-modal supervision.

We agree that the original presentation did not make the individual effects of TFA and dual-α sufficiently clear. In our original ablation experiments, we had already evaluated the complete semantic framework using the original TS-FSAR relation formulation without dual-α enhancement, but omitted this result from our previous response:

ConfigurationAccuracy (%)
SLR + SVF, without TFA or dual-α61.06
TFA + SLR + SVF, without dual-α61.63
TFA + SLR + SVF, with dual-α61.99

Under the same original relation formulation and with SLR and SVF fixed, adding TFA improves accuracy from 61.06% to 61.63%, isolating an incremental contribution of 0.57 percentage points from TFA. Keeping TFA, SLR, and SVF fixed and then enabling dual-α further improves accuracy from 61.63% to 61.99%, contributing an additional 0.36 percentage points. These results show that TFA remains effective without dual-α, while dual-α provides a separate complementary improvement. We will add the previously omitted 61.63% control to Table 2 and revise the discussion to report the two contributions separately.

Reviewer_3xhKOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Semantic-Adaptive Relation Fusion (SARF) for few-shot action recognition, building on the CLIP-based TS-FSAR framework. It adds text-guided feature alignment (TFA), two residual adapters for semantic logits recalibration (SLR), and instance-dependent semantic–visual fusion (SVF) for auxiliary supervision. A dual-alpha relation representation is also used. At inference, predictions from the relation, CLIP semantic, and LSN semantic branches are combined using fixed weighted-product fusion. Experiments cover four datasets with five benchmark splits under 5-way 1-shot and 5-shot settings. The reported gains over the reproduced baseline are largest on SSv2, particularly in the 1-shot setting.

Critical strengths

The evaluation covers several standard benchmarks and includes a reproduced TS-FSAR baseline, component ablations, and comparisons of alternative designs. The reported improvements on SSv2 are worth investigating. This breadth of evaluation provides a useful basis for assessing the proposed training additions, although the concerns below limit how confidently their effects can currently be interpreted.

Critical weaknesses
  1. The methodological advance is incremental, and a central training mechanism is insufficiently specified. The framework builds on TS-FSAR, while residual correction of CLIP logits has precedent in Logits DeConfusion. The value of the new combination therefore depends on demonstrating how its training additions improve the retained representations. In particular, Table 2(a) reports an improvement from 59.99% to 60.64% for SLR alone, without SVF. According to Sections 3.1–3.5 and Figure 2, SLR receives logits from frozen CLIP features and is omitted at inference. The path by which this isolated branch affects inference-time parameters is not specified. SLR can influence the visual network through SVF in the full model, but that does not explain this ablation. Appendix B lists inherited losses without resolving whether recalibrated logits enter a KL term or another trainable connection. This gap prevents a clear interpretation of the claimed module contribution.
  2. The evidence does not yet establish reliable gains or isolate the benefit of instance-dependent fusion. Appendix B reports only training seed 0. Several improvements over the reproduced baseline are small, including 0.1–0.3 percentage points on Kinetics-100 and UCF-101; averaging 10,000 evaluation episodes does not measure variability across training runs. The larger SSv2-Small gain is encouraging, but also needs uncertainty estimates. Table 2(d) does not specify the fixed fusion weight or its selection procedure. Given that the learned weights strongly favor visual evidence (Table 3), comparison with a well-tuned fixed weight or a globally learned scalar is needed to isolate the value of per-instance weighting. The performance comparison also omits TSAM (Li et al., AAAI 2025), which reports 97.1% on Kinetics 5-shot versus SARF’s 96.8% with the same CLIP backbone family and standard few-shot setting. Unless a protocol difference justifies excluding it, the claim of best performance in nine of ten settings needs revision.
Questions for authors
  1. In the SLR-only ablation without SVF, which parameters retained at inference are affected by SLR training? Please specify the complete connection to the base losses, including any KL term or trainable video adapter, and explain the 59.99% to 60.64% result under otherwise matched settings.
  2. Are the gains consistent across independent training seeds? Please report the mean and variation of the improvement over a baseline trained under the same configuration, preferably evaluated on common episodes.
  3. What fixed fusion weight was used in Table 2(d), and how was it selected on validation data? Does per-instance SVF still help relative to a sufficiently tuned fixed weight or a globally learned scalar?
  4. Why is TSAM absent from Table 1? Please clarify comparability with its reported Kinetics 5-shot result and update the comparison and state-of-the-art claim if the protocols are comparable.
2 comments
AuthorsOpenReview ↗

Response to Reviewer 3xhK

Thank you for the constructive comments. We have clarified the methodological contribution and the training-to-inference role of SLR, and added robustness and comparison evidence.

We respectfully clarify that the contribution is not the isolated use of attention or residual adapters, but their coordinated use at three distinct stages of episodic action recognition. TFA introduces video-conditioned text tokens before relation construction, allowing semantics to shape local visual relations. SLR adapts transferred semantic responses through training-time residual supervision while improving the semantic pathway retained at inference. SVF then connects the two pathways through query-dependent semantic-visual supervision. In contrast to methods that introduce text mainly at the prototype or prediction level, SARF coordinates semantics during relation formation, response adaptation, and cross-modal supervision. The component ablations further show that these mechanisms provide complementary rather than redundant improvements.

The frozen CLIP encoder produces frame-level features, which are processed by the learnable visual adapter inherited from TS-FSAR and temporally averaged into FclipF_{clip}. Matching this representation with the text embeddings yields SclipS_{clip}. During training, SclipS_{clip} enters SLR without detachment, so LslrL_{slr} updates both the residual SLR adapters and the inherited visual adapter. At inference, the SLR adapters and SslrS_{slr} are removed, but the optimized visual adapter and its SclipS_{clip} remain in the final weighted-product prediction. The inherited KL term aligns the training-category visual classification logits with the raw CLIP semantic logits SclipS_{clip}; SslrS_{slr} does not enter this objective. Hence, the SLR-only gain arises from training-time supervision of the retained semantic pathway, rather than from test-time use of SslrS_{slr} or an indirect KL connection. This connection is now explicit in Sections 3.1, 3.3, and 3.5.

We repeated both TS-FSAR and SARF with three independent training seeds (0, 1, and 2) on SSv2-Small and HMDB-51. For each dataset, all checkpoints were evaluated on the same 10,000 test episodes generated using a fixed evaluation seed:

DatasetMethodSeed 0Seed 1Seed 2Mean ± std.
SSv2-SmallTS-FSAR59.9960.3159.8460.05 ± 0.24
SSv2-SmallSARF61.9961.8061.0361.61 ± 0.51
HMDB-51TS-FSAR85.0484.5785.0384.88 ± 0.27
HMDB-51SARF85.6785.7885.4385.63 ± 0.18

SARF improves over TS-FSAR in every paired run, with average gains of 1.56 ± 0.41 percentage points on SSv2-Small and 0.75 ± 0.42 percentage points on HMDB-51. These results show that the improvement is consistent across training seeds and is not specific to a single dataset.

For fusion, Fixed Average uses γ = 0.5, and Fixed Weighted uses γ = 0.7 for visual logits and 0.3 for recalibrated semantic logits. We further added a learned global scalar under the same setting:

Fusion strategyγAccuracy (%)
Without SVF--61.55
Fixed average0.5060.48
Fixed weighted0.7059.60
Learned global scalar0.8761.32
Instance-dependent SVFquery-dependent61.99

The learned global alternative is stronger than the manually fixed variants, but remains below instance-dependent SVF, supporting the value of query-specific semantic-visual balancing. These fusion strategies affect auxiliary training supervision only; the final inference protocol is unchanged.

Finally, we thank the reviewer for identifying TSAM, which was inadvertently omitted from our original literature comparison. We have now added it to Table 1. Under the comparable CLIP ViT-B/16 setting, TSAM reports 96.2%/97.1% on Kinetics-100 1-shot/5-shot, while SARF obtains 96.4%/96.8%. SARF is therefore slightly lower by 0.3 percentage points in the 5-shot setting, but remains higher in the 1-shot setting and achieves leading results in eight of the ten evaluated settings overall. We have corrected the corresponding claim from nine of ten to eight of ten settings.

AuthorsOpenReview ↗

Rebuttal follow-up

Dear Reviewer,

I hope this message finds you well.

As the discussion period is drawing to a close, we would like to respectfully check whether our rebuttal has helped address the initial concerns you raised. Please let us know if you have any further questions or if there is anything else we can clarify. We would be more than happy to discuss further. Your perspective is crucial for us to improve the paper.

Thank you again for your time and valuable review!

Best regards!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.