acceptodds
Withdrawn from ICLR 2027

TASA: TOKEN-ADAPTIVE STEERING OF ACTIVATIONS FOR VIDEO TEMPORAL GROUNDING

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Video temporal grounding (VTG) aims to identify the start and end timestamps of the video segment corresponding to a natural-language query, yet existing Video-LLMs often produce inaccurate temporal boundaries. Activation steering guides model behavior by modifying internal activations during inference. It offers a promising way to improve temporal grounding without updating model parameters. Conventional methods typically derive a single shared direction from positive-negative examples and apply it uniformly to all test samples. However, VTG involves diverse and even opposing boundary errors, making positive-negative pairs ambiguous and a single global direction insufficient for sample- and position-specific correction. We propose Token-Adaptive Steering of Activations (TASA), which uses three-stage training to distill task-loss gradients into a compact bank of steering vectors and learns a lightweight controller consisting of a token-level Router and Gate, thereby extending conventional single-direction steering to adaptive multi-directional activation steering. At inference, the controller adaptively composes the vectors and scales the intervention at each autoregressive generation position according to the current hidden state. This design combines the efficiency of shared activation steering with the flexibility of token-specific directions. Extensive experiments across multiple open-source Video-LLMs and widely used temporal grounding benchmarks show that TASA consistently delivers substantial improvements without updating the Video-LLM backbone parameters. In particular, on Charades-STA with Qwen3-VL-8B, TASA raises mIoU from 43.92 to 56.27 (+12.35) and [email protected] from 45.13 to 66.18 (+21.05). These results demonstrate that task-loss gradients can be distilled into compact, learnable activation-space interventions that effectively improve Video-LLM temporal grounding.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Video Temporal Grounding, Multimodal Large Language Models, Activation Steering, Hidden-State Intervention, Token-Adaptive Control

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_kMGqOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

TASA proposes token-adaptive activation steering to improve video temporal grounding without updating the Video-LLM backbone. It extracts local correction directions from negative gradients of the timestamp prediction loss. Spherical K-means compresses these directions into a compact steering-vector bank. A token-level Router dynamically combines the vectors based on the current hidden state. A Gate controls the intervention strength during autoregressive decoding. The method uses three-stage training to initialize and refine the vector bank and controller. Experiments across five Video-LLMs and three benchmarks show consistent improvements, including a 12.35-point mIoU gain on Charades-STA with Qwen3-VL-8B.

Critical strengths
  1. Consistent performance improvements. TASA improves temporal grounding across all 15 model–dataset settings, including cross-dataset evaluation on QVHighlights.

  2. Parameter-efficient adaptation. The method keeps the backbone frozen and optimizes only a small intervention module with approximately 35K–70K trainable parameters.

  3. Useful empirical analysis. Ablations examine vector initialization, bank size, and intervention depth, providing evidence for the effectiveness of gradient-derived multi-vector steering.

Critical weaknesses
  1. Insufficient comparison with alternative steering methods. Gradient-derived directions are compared mainly against random initialization, without matched contrastive-steering baselines. The gains over single-layer LoRA are also marginal (0.14–0.74 mIoU).

  2. Unclear consistency and roles of steering directions. The paper lacks analysis of vector stability across runs and their correspondence to specific boundary errors, such as early/late starts or ends.

  3. The paper does not distinguish improvements from correcting temporal boundaries of correctly identified events versus recovering from incorrect event localization.

  4. Incomplete component ablations. The individual contributions of the Gate and regularization losses remain unclear. The necessity of the three-stage optimization procedure is also insufficiently validated.

  5. Limited generalization evaluation. More challenging multi-segment grounding (OMTG-Bench) and general video understanding benchmarks (Video-MME, Video-MME-v2) are missing.

  6. Insufficient efficiency analysis. The paper reports trainable parameter counts but lacks end-to-end training time, GPU memory, and inference latency comparisons with LoRA.

Questions for authors

See Weaknesses.

Reviewer_x4HWOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes TASA, a lightweight activation-steering method for improving temporal grounding in frozen Video-LLMs. Unlike conventional steering methods that rely on contrastive examples, TASA derives steering directions from answer-loss gradients at timestamp-prediction positions. These directions are clustered into a small vector bank, and a token-dependent router and gate dynamically determine the steering intervention. The module is trained in two stages: first optimizing the router and gate with the steering vectors frozen, followed by joint refinement.

Experiments across five Video-LLM backbones and three temporal grounding benchmarks (Charades-STA, ActivityNet, and QVHighlights) show consistent improvements over the authors’ baseline inference setup, with performance comparable to single-layer LoRA in the reported comparison.

Critical strengths

The paper is well organized. Deriving steering directions from answer-loss gradients is a reasonable alternative to contrastive steering, particularly for structured outputs such as temporal boundaries, where constructing meaningful contrastive examples can be challenging. TASA consistently improves performance across five backbones and three datasets while introducing a relatively lightweight trainable module.

Critical weaknesses

Although presented as activation steering, TASA involves supervised optimization of a trainable intervention module, making its distinction from existing representation-level adaptation and parameter-efficient fine-tuning methods somewhat unclear. Closely related approaches, including ReFT/LoReFT, RED, LoFiT, and MoV, are neither adequately discussed nor experimentally compared. These methods share important design principles with TASA, such as learning low-dimensional representation interventions or combining multiple steering directions. Moreover, the reported comparison with single-layer LoRA shows nearly identical performance in the main Qwen3-VL-8B setting (56.27 vs. 56.13 mIoU). Stronger comparisons under matched training and parameter budgets are necessary to establish whether TASA offers a meaningful methodological advantage beyond existing lightweight adaptation techniques.


The reported improvements are substantial, but the baseline evaluation protocol differs from the official evaluation settings of the underlying models. For example, the paper evaluates Qwen3-VL using 2 fps and a custom prompting format, whereas the Qwen3-VL technical report evaluates Charades-STA at 4 fps with a different prompt and reports substantially higher baseline performance. In particular, the official Qwen3-VL-8B result (56.0 mIoU) is already comparable to TASA’s reported performance (56.27), while the paper’s baseline achieves only 43.92. Although these results are not directly comparable due to the different evaluation protocols, this discrepancy raises concerns about how much of the reported improvement is attributable to TASA rather than the baseline configuration. Evaluating both the baseline and TASA under the models’ official protocols would substantially strengthen the empirical claims.

Questions for authors
  1. Could the authors evaluate both Base and TASA under the official temporal grounding protocols of the underlying models, particularly Qwen3-VL (4 fps and the official prompting format)? Please also clarify whether the Instruct or Thinking variant is used. This comparison is particularly important for determining the magnitude of improvements attributable to TASA.

  2. How does TASA compare with existing representation-level adaptation methods such as LoReFT, RED, or a conventional bottleneck adapter under matched training data, intervention layers, and parameter budgets? It would also be helpful to extend the single-layer LoRA comparison to additional backbones.

  3. Could the authors provide results across multiple training seeds and investigate whether simpler calibration approaches can recover some of the reported improvements? For example, fitting a lightweight mapping from the baseline’s predicted temporal boundaries to ground-truth intervals would help clarify whether TASA primarily improves temporal understanding or corrects systematic prediction biases.

  4. Could the authors report standard QVHighlights metrics (e.g., [email protected], [email protected], and mAP) and clarify how the evaluation handles multiple annotated intervals? Additionally, an analysis of the learned routing and gating behavior across different error types would help substantiate the motivation for token-adaptive steering.

1 comment

The authors have not responded. All three reviewers find that TASA performs roughly on par with single-layer LoRA and question what it offers beyond existing adapters. My concern about the Base evaluation protocol, where Base scores substantially below the official Qwen3-VL results, also remains unaddressed. I maintain my rating of 2.

Reviewer_zsyXOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes TASA (Token-Adaptive Steering of Activations), an activation steering framework for improving video temporal grounding (VTG) in Video Large Language Models (Video-LLMs) without modifying their pretrained backbone parameters. Unlike conventional methods that rely on a single steering direction derived from positive–negative activation pairs, TASA extracts task-specific correction directions from temporal grounding loss gradients and compresses them into a compact vector bank. A lightweight token-level Router and Gate dynamically combine these vectors and adjust intervention strength based on the current hidden state. Experiments across multiple Video-LLMs and VTG benchmarks demonstrate consistent improvements in temporal localization accuracy, highlighting the effectiveness of adaptive activation steering for parameter-efficient VTG adaptation.

Critical strengths

1. Novel and well-motivated approach to video temporal grounding.
This paper explores a novel application of activation steering to video temporal grounding, which remains relatively underexplored in the literature. The motivation is well justified, as diverse temporal boundary errors may require different correction directions, making conventional single-vector steering potentially insufficient. The proposed gradient-derived vector bank and token-adaptive routing provide a reasonable approach to addressing this challenge.


2. Parameter-efficient adaptation without modifying the pretrained backbone.
TASA improves temporal grounding by learning a lightweight steering module while keeping the Video-LLM backbone frozen. This provides an interesting alternative to conventional task-specific fine-tuning and demonstrates the potential of activation-space interventions for improving temporal localization.


3. Extensive experiments demonstrating consistent improvements.
The paper evaluates TASA across five Video-LLM backbones and three VTG benchmarks, demonstrating consistent improvements over the corresponding baselines. In particular, the improvements on QVHighlights, which is excluded from training, provide encouraging evidence of cross-dataset generalization. The ablation studies further support the effectiveness of gradient-derived vector initialization, multiple steering directions, and adaptive routing.

Critical weaknesses

1. Limited analysis of how learned steering directions correct temporal boundary errors.

TASA derives steering directions from token-level cross-entropy gradients, which encourage correct token prediction but do not explicitly capture numerical differences between timestamps. While the ablation studies demonstrate the benefits of multiple steering vectors and adaptive routing, the paper provides limited interpretability analysis of how these vectors correspond to different temporal correction behaviors. In particular, it remains unclear whether individual vectors specialize in correcting early or late boundary predictions, or whether distinct routing patterns emerge for start and end timestamps. Further analysis of vector-specific correction behaviors and token-level routing patterns would better substantiate the motivation behind the proposed multi-vector steering framework.


2. Unclear practical advantages over simpler parameter-efficient adaptation methods.

TASA introduces a relatively complex three-stage training procedure involving gradient-based direction discovery, controller optimization, and joint refinement with multiple regularization terms. However, the comparison with single-layer LoRA in Appendix Table 5 shows largely comparable performance under similar trainable-parameter budgets. Although TASA preserves the pretrained backbone parameters, the paper does not provide a systematic comparison of training cost, memory consumption, or inference overhead. Consequently, it remains unclear whether the additional methodological complexity offers sufficient practical advantages over simpler parameter-efficient adaptation approaches.

Questions for authors
  1. Do the learned steering vectors exhibit distinct correction behaviors, such as adjusting early versus late boundary predictions or start versus end timestamps? Could the authors provide a qualitative analysis or visualization of vector-specific correction patterns?

  1. Since the steering directions are derived from token-level CE gradients rather than explicit temporal errors, could the authors analyze how the resulting interventions affect the magnitude and direction of temporal boundary errors?

  1. Given the comparable performance between TASA and single-layer LoRA in Appendix Table 5, could the authors clarify the practical advantages of the proposed three-stage framework, particularly in terms of training time, GPU memory consumption, and inference overhead?

  1. Although TASA demonstrates promising cross-dataset generalization on QVHighlights, the steering vectors and controller are trained using only Charades-STA and ActivityNet Captions. Given the diversity of video domains, query types, and temporal error patterns, how well can the learned steering vector bank cover unseen scenarios? Could the authors provide further analysis of how training data diversity affects the generalization of the learned steering directions?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.