acceptodds
Withdrawn from ICLR 2027

Reliability-Aware Test-time Reinforcement Learning for Anomalous Video Understanding

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Anomalous video understanding aims not only to identify abnormal events in videos, but also to interpret their objects, actions, interactions, and high-level semantics. Recent Video Large Language Models (Video-LLMs) have demonstrated promising zero-shot capabilities for this task, yet frozen models remain limited by diverse anomaly patterns and evolving deployment environments. Test-time reinforcement learning offers a promising solution by enabling models to adapt from self-generated feedback without additional human annotations. However, directly applying it to anomalous video understanding faces three reliability bottlenecks: (1) pseudo-labels may be unstable across semantically equivalent queries, introducing unreliable self-supervision; (2) coarse consensus rewards ignore generation uncertainty, weakening reward informativeness; and (3) unanimous rollout groups receive identical rewards, causing group-relative advantages to collapse and eliminating effective policy-gradient signals. To address these challenges, we propose a reliability-aware online test-time reinforcement learning framework that improves reliability at the sample, generation, and optimization levels. Specifically, dual-query consistency filtering retains samples with cross-query consistent pseudo-labels, an entropy-aware consensus reward combines answer agreement with generation uncertainty, and virtual-anchor advantage estimation preserves informative relative advantages for unanimous rollout groups. Experiments on VAU-Bench demonstrate consistent improvements over frozen and supervised baselines. Notably, on the ECVA subset with thinking, our method improves accuracy from 75.81% to 90.18% over the frozen backbone.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Anomalous Video Understanding, Test-time Reinforcement Learning, Video Large Language Models, Test-time Adaptation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_zVQdOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces a reliability-aware test-time reinforcement learning (TTRL) framework for anomalous video understanding (AVU) with Video-LLMs. It addresses three limitations of existing TTRL methods: inconsistent pseudo-labels for semantically equivalent queries, reward estimation that overlooks generation uncertainty, and vanishing GRPO advantages when all rollouts agree. To address these issues, the framework combines dual-query consistency filtering, an entropy-aware consensus reward, and a virtual negative anchor for advantage normalization. Experiments on VAU-Bench with Qwen2.5-VL-3B show consistent improvements over frozen and supervised baselines in multiple-choice question answering. The method also improves anomaly classification performance without additional task-specific supervision.

Critical strengths

1.The paper addresses several practical limitations of test-time reinforcement learning through three complementary designs. The method improves pseudo-label reliability through dual-query consistency filtering, provides finer-grained rewards by incorporating prediction uncertainty, and mitigates GRPO advantage collapse without additional rollouts.

2.The experiments are comprehensive, covering multiple standard distribution-shift benchmarks, comparisons with several baselines, and ablation studies to assess the contribution of each component.

Critical weaknesses
  1. The core idea of using reliability or confidence measures for sample selection in test-time adaptation (TTA) is not new. EATA[1] explicitly selects reliable samples based on prediction entropy and filters redundant samples using prediction similarity. Similarly, SAR[2] employs entropy-based reliable sample filtering to mitigate unstable model updates. However, the paper does not clearly distinguish its proposed reliability metric from these existing approaches or sufficiently justify its novelty.
  2. The motivation is not sufficiently clear. While the Introduction discusses the advantages of test-time adaptation (TTA) for anomalous video understanding, it does not adequately explain why test-time reinforcement learning (TTRL) is particularly suitable for this task or what limitations of existing TTA methods motivate its use.

[1] Niu, S., Wu, J., Zhang, Y., Chen, Y., Zheng, S., Zhao, P., and Tan, M. Efficient Test-Time Model Adaptation without Forgetting. Proceedings of the 39th International Conference on Machine Learning (ICML), pp. 16888–16905, 2022. [2] Niu, S., Wu, J., Zhang, Y., Wen, Z., Chen, Y., Zhao, P., and Tan, M. Towards Stable Test-Time Adaptation in Dynamic Wild World. International Conference on Learning Representations (ICLR), 2023.

Questions for authors
  1. How are the thresholds for distinguishing reliable and unreliable samples determined? How sensitive is the proposed method to different threshold values?
  2. How does the proposed method compare with more recent TTA/TTRL approaches?
  3. Did you experiment with simpler pseudo-label self-training using cross-entropy on the majority label (without GRPO)? A comparison would help isolate the value of group-normalized policy-gradient updates versus supervised pseudo-labeling.
5 comments

We sincerely thank the reviewer for the thoughtful feedback and constructive suggestions on the motivation, methodological distinctions, and experimental comparisons.

Weakness 1. The core idea of using reliability or confidence measures for sample selection in test-time adaptation (TTA) is not new. EATA[1] explicitly selects reliable samples based on prediction entropy and filters redundant samples using prediction similarity. Similarly, SAR[2] employs entropy-based reliable sample filtering to mitigate unstable model updates. However, the paper does not clearly distinguish its proposed reliability metric from these existing approaches or sufficiently justify its novelty.

We agree that reliability-based sample selection has been studied in existing TTA methods. Our contribution concerns how reliability is assessed and used in generative anomalous-video understanding. We clarify the differences from EATA and SAR below.

(1) Task formulation. EATA and SAR address discriminative prediction, whereas our method adapts a generative video-language model for anomalous-video question answering. Because generated responses can express the same answer through different wording or reasoning paths, we assess agreement at the answer level across generated responses and question formulations.

(2) Reliability criterion. EATA selects low-entropy predictions and filters redundant samples based on prediction similarity, while SAR uses entropy-based reliable sample filtering. In contrast, our method retains a sample only when the original and semantically equivalent reformulated questions produce the same strict-majority answer. Because low-entropy predictions can still disagree across question formulations, confidence alone does not establish cross-query consistency. Our criterion therefore explicitly checks whether the predicted answer remains stable under question reformulation.

(3) Role in optimization. EATA and SAR select reliable predictions for entropy minimization. In our framework, cross-query consistency instead selects response groups that provide reliable answer-level feedback for reward construction. Furthermore, token entropy is converted into confidence weights and aggregated by answer to construct rewards, which are then used for response-level policy optimization. Therefore, cross-query consistency determines which groups provide supervision, while token entropy contributes to the rewards used to optimize the generative policy.

In summary, our contribution lies in assessing reliability through cross-query answer consistency and using the retained answer-level feedback to construct rewards for test-time adaptation in generative anomalous-video understanding. We will clarify the differences in task formulation, reliability criterion, and optimization objective from EATA and SAR in the revised manuscript.

Weakness 2. The motivation is not sufficiently clear. While the Introduction discusses the advantages of test-time adaptation (TTA) for anomalous video understanding, it does not adequately explain why test-time reinforcement learning (TTRL) is particularly suitable for this task or what limitations of existing TTA methods motivate its use.

Our task uses a video-language model to perform reasoning about anomalous videos. However, neither ground-truth answers nor reference reasoning sequences are available at test time. Therefore, adaptation requires an optimization objective that can use feedback derived from the model’s own responses without relying on annotated target sequences.

EATA and SAR use entropy-minimization-based objectives to increase prediction confidence on selected samples. However, when applied to generation, entropy minimization optimizes how certain the output distribution is, whereas the task requires correct answers and sound reasoning. Greater certainty alone does not establish either answer correctness or reasoning quality. In contrast, TTRL converts response rewards into relative advantages for policy optimization, allowing higher-reward responses to be reinforced and lower-reward responses to be suppressed when sampled answers disagree. Therefore, we adopt TTRL to use answer-level feedback derived from repeated sampling and cross-query consistency to construct rewards and guide response-level optimization.

In summary, TTRL is suitable for the task because it uses self-generated answer-level feedback to optimize complete responses without reference answer or reasoning annotations, while allowing relative rewards to distinguish competing responses. We will make the task-specific motivation explicit in the revised Introduction.

Q1. How are the thresholds for distinguishing reliable and unreliable samples determined? How sensitive is the proposed method to different threshold values?

(1) We select reliable samples through strict-majority agreement across two query views, rather than an entropy threshold. Specifically, we set the answer-support threshold to 0.5, requiring an answer to receive more than half of the NN responses in each query view. The majority answers from the original and reformulated queries must also match for the sample to be retained. Because a strict majority gives the selected answer more support than all alternatives combined, while cross-query agreement checks whether the answer remains stable under semantically equivalent reformulation, we combine both conditions to select reliable pseudo-supervision. The same rule is applied across all datasets without dataset-specific threshold tuning. The parameters τ\tau and δ\delta are used for reward redistribution, not for determining whether a sample is accepted.

(2) To examine sensitivity to different rollout counts and their corresponding majority requirements, we compare N=2,3,4N=2,3,4 on MSAD under w/o Think, with batch size fixed at 2. The minimum required votes per query view are 2, 2, and 3, respectively, corresponding to agreement levels of 100%, 66.67%, and 75%. All configurations additionally require the majority answers across the two query views to match. The table reports the rollout count, required votes, minimum agreement level, and resulting QA accuracy. The results are shown below.

Rollouts per viewRequired votes per viewMinimum agreementQA accuracy
N=2N=22100%92.50%
N=3N=3266.67%92.50%
N=4N=4375%91.67%

As shown in the table, compared with the frozen backbone’s accuracy of 85.83%, the three configurations improve accuracy by 6.67, 6.67, and 5.84 percentage points, respectively. Furthermore, N=2N=2 and N=3N=3 achieve identical accuracy, while N=4N=4 is only 0.83 percentage points lower. Therefore, the method consistently outperforms the frozen backbone and exhibits limited performance variation across the evaluated rollout counts and corresponding majority requirements.

In summary, sample selection uses a fixed cross-query strict-majority rule rather than a dataset-specific entropy threshold. The results demonstrate stable performance across the evaluated sampling and consensus configurations.

Q2. How does the proposed method compare with more recent TTA/TTRL approaches?

To evaluate our method against recent test-time reinforcement learning approaches, we have already included TTRL (NeurIPS 2025) [3] in Table 3, corresponding to the Binary 0/1 reward baseline. We additionally compare with TTRV (CVPR 2026) [4] in the supplementary experiments.

To maintain consistent experimental settings, all three methods use Qwen2.5-VL-3B-Instruct, three sampled responses per query view, and a batch size of 2. For each batch, all methods perform one parameter update using the unlabeled inputs and then immediately evaluate the updated model on the same batch. Updated parameters are retained for subsequent batches. All methods follow the same evaluation protocol, including video preprocessing, prompts, visual-token budget, and final evaluation decoding settings.

We evaluate all methods on UCF-Crime, MSAD, and ECVA under both w/o Think and w/ Think. Each row corresponds to one method, and the columns report QA accuracy (%) for each dataset and reasoning setting. The results are presented in the table below.

MethodUCF-Crime w/o ThinkUCF-Crime w/ ThinkMSAD w/o ThinkMSAD w/ ThinkECVA w/o ThinkECVA w/ Think
TTRL91.6386.0687.0885.4286.0785.16
TTRV92.0390.8487.9287.5089.2789.04
Ours92.8392.4392.5090.0091.5589.95

As shown in the table, compared with TTRL, our method improves accuracy in all six settings by 1.20–6.37 percentage points, with an average improvement of 4.64 percentage points. Furthermore, compared with TTRV, our method achieves improvements of 0.80–4.58 percentage points, averaging 2.11 percentage points across the six settings. Therefore, the performance gains hold across all three datasets and both reasoning settings.

In summary, our method consistently outperforms TTRL and TTRV with the same backbone, per-view rollout count, batch size, and evaluation protocol, demonstrating its effectiveness relative to recent test-time reinforcement learning approaches.

[3] TTRL: Test-Time Reinforcement Learning. NeurIPS, 2025.

[4] TTRV: Test-Time Reinforcement Learning for Vision Language Models. CVPR, 2026.

Q3. Did you experiment with simpler pseudo-label self-training using cross-entropy on the majority label (without GRPO)? A comparison would help isolate the value of group-normalized policy-gradient updates versus supervised pseudo-labeling.

(1) To explain the distinction between the optimization objectives, we consider the composition of retained rollout groups. For unanimous groups, equal positive advantages give the policy update a positive self-training interpretation. However, when retained groups contain competing answers, full-response SFT increases the likelihood of every training response, whereas group-relative advantages enable the policy objective to reinforce higher-reward responses and suppress lower-reward responses. Therefore, advantage-weighted optimization can use answer disagreement to provide a discriminative learning signal rather than positively fitting all retained responses.

(2) Following the reviewer’s suggestion, we conduct an additional self-training experiment on MSAD with N=3N=3 to evaluate the value of advantage-weighted policy optimization over direct pseudo-label self-training. We also retain KL regularization in both methods to constrain deviation from the reference policy and limit excessive model updates. For the self-training control, we replace the advantage-weighted policy objective with token-level cross-entropy over the complete generated responses, including reasoning tokens where present, rather than supervising only the majority-answer label.

In the table, Full-response SFT + KL denotes the direct self-training control, and Ours denotes our adaptation method. The two columns report QA accuracy (%) under w/o Think and w/ Think, respectively. The results are reported below.

Methodw/o Thinkw/ Think
Full-response SFT + KL90.8387.92
Ours92.5090.00

As shown in the table, our method improves accuracy over full-response SFT + KL by 1.67 and 2.08 percentage points under w/o Think and w/ Think, respectively. Therefore, even with the same generated supervision and KL regularization, directly fitting the responses does not achieve the accuracy of our adaptation method.

Furthermore, we observe degeneration of explicit reasoning output in the w/ Think SFT + KL run. Among 240 predictions, only 111 contain a <think> reasoning section. Starting at step 53, the remaining 129 predictions contain only <answer>X</answer>, despite the instruction to generate reasoning. Because the self-training targets include the reasoning tokens present in the generated responses, the degeneration cannot be attributed simply to answer-only supervision. Thus, fitting self-generated texts containing reasoning does not by itself ensure that explicit reasoning output is maintained during continual adaptation.

In summary, our method outperforms full-response self-training under matched generated supervision and KL regularization. Unlike uniformly reinforcing the training responses, advantage-weighted policy optimization can additionally use answer disagreement to distinguish the update directions of competing responses. The accuracy results support the practical value of the optimization approach over direct pseudo-label fitting.

Reviewer_MztUOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies online adaptation of video LLMs using unlabeled test videos. It filters adaptation samples by checking consensus between the original question and a semantically equivalent reformulation, constructs entropy-weighted consensus rewards, and adds a virtual negative anchor to avoid zero advantages for unanimous groups. The model is updated with a KL-regularized group-relative objective.

Across three VAU-Bench subsets and two prompting settings, QA accuracy improves over frozen Qwen2.5-VL-3B and matches or exceeds the supervised VAU-R1 model. On ECVA with thinking, accuracy rises from 75.81% to 90.18%. The appendix provides prompts, answer-parsing rules, and failure cases.

Critical strengths
  1. Substantial gains across all evaluated QA settings. The method improves over the frozen backbone on all three subsets under both prompting settings. The 14.37-percentage-point gain on ECVA with thinking makes the approach worth investigating.
  2. Clear adaptation protocol and useful failure analysis. The paper specifies adaptation before prediction, model resets between subsets, query reformulation, and answer parsing. The QA-adapted model improves several classification results without further updates. The appendix also illustrates how shared visual blind spots can sustain incorrect answers across both query views.
Critical weaknesses
  1. Entropy weighting does not distinguish rewards in the default setting, and the anchor has not been separated from simple self-training.

    Section 4.1 on pp. 6–7 uses N=2N=2 trajectories per query view, while Eq. (4) requires a strict majority in both groups and agreement between them. Both accepted original-query trajectories must therefore predict the same valid answer. Equation (8) normalizes only over observed answers, giving p0=1p_0=1; Eq. (11) preserves this distribution, and Eq. (12) assigns reward 1 to both trajectories. Changes in generation entropy cannot affect these rewards.

    Using the sample standard deviation in Appendix Algorithm 1, adding a virtual reward of 0 to real rewards (1,1)(1,1) gives both real trajectories an advantage of approximately sva/3s_{\mathrm{va}}/\sqrt{3}, ignoring the denominator's 10−1210^{-12} term. This produces a learning signal but does not rank trajectory quality. For a single on-policy update, it is closer to reinforcing filtered self-generated text with KL regularization. Table 3 removes both dual-query filtering and the anchor, and the successful case study uses N=3N=3, so neither directly tests entropy weighting in the default full configuration.

    Please compare the default model with a variant that retains filtering and the anchor but fixes rewards at 1, and with self-training plus KL on the same generated text. The value of svas_{\mathrm{va}} should also be reported. If the implementation uses different filtering or trajectory-level rewards, the equations and configuration need to reflect that.

  2. The contributions of additional inference and parameter updates remain unclear.

    The method generates an additional video description, reformulates the question, and samples four responses per example. The main comparisons are against single-pass frozen predictions and offline supervised models. The binary-reward control in Table 3 does not fully separate reformulation, voting, and adaptation. Although TTRL, TTRV, and TTA-Vid are discussed, comparisons with the closest alternatives on the same backbone and task are missing. The contribution should also be distinguished from confidence weighting in SCOPE and virtual-sample designs in AVSPO.

    A useful control would use the same two query views and generation budget for voting without parameter updates. Comparisons with basic test-time RL and filtered self-training on the same backbone would then establish the added value of adaptation.

  3. The evidence mainly supports one backbone on shuffled test streams.

    The experiments use one 3B backbone, with separate resets and shuffled ordering for the three subsets. A stable cumulative accuracy curve does not establish behavior under scene changes or consecutive difficult examples. Table C.2 links cross-query disagreement to errors, but does not fully report accepted pseudo-label accuracy, coverage, or the proportion of consistently wrong answers. The appendix failure case shows that both query views can share a visual blind spot.

    Please clarify whether QA adaptation and classification evaluation reuse the same source videos: cross-task gains and generalization to unseen videos are different conclusions. Hyperparameter selection, end-to-end adaptation time, and memory use also need to be specified. Claims about long-term deployment should match the evaluated setting.

Questions for authors
  1. With the default N=2N=2, do all accepted original-query trajectories receive reward 1? Where does entropy affect the actual update? What changes when rewards are fixed at 1 while retaining filtering and the anchor?
  2. What is svas_{\mathrm{va}}? Does the anchor improve over simple self-training with the same samples, generated text, and KL regularization?
  3. How do cross-query voting without updates and basic test-time RL perform under the same generation budget? Can the frozen backbone be evaluated with the current inference implementation?
  4. What are the coverage and pseudo-label accuracy of the accepted and rejected groups? Do QA adaptation and classification evaluation share source videos?
  5. What unlabeled criterion or independent validation information is used for hyperparameter selection? What are the per-video adaptation time, total generation cost, and peak memory use?
6 comments

We sincerely thank the reviewer for the careful examination of our method and evaluation protocol, and for the constructive suggestions that have helped us improve the rigor and clarity of the paper.

Q1. With the default N=2N=2, do all accepted original-query trajectories receive reward 1? Where does entropy affect the actual update? What changes when rewards are fixed at 1 while retaining filtering and the anchor?

(1) Under the default N=2N=2, all accepted original-query trajectories receive reward 1. Consequently, entropy does not affect the final rewards or the resulting update through reward weighting. Therefore, replacing the entropy-aware rewards with constant rewards of 1, while retaining the same filtering and virtual anchor, leaves the reward and advantage calculations unchanged and produces the same parameter updates when all other conditions are identical.

Nevertheless, we consider equal answer-level rewards appropriate because responses supporting the same cross-query-consistent answer share the same supervision target. Furthermore, the reliability experiments reported in our response to Reviewer 1, Q1 demonstrate that consistency filtering improves pseudo-label accuracy. Therefore, responses supporting the same retained answer need not receive different answer-level rewards solely because their generation confidence differs.

(2) Entropy matters when sampled responses contain diverse answers. Specifically, entropy adjusts how strongly competing responses are reinforced or suppressed: when retained responses point to different answers, we convert each response's mean token entropy into a confidence weight and aggregate these weights by answer. As a result, the relative support for each answer depends on both response frequency and generation confidence, rather than frequency alone. To examine entropy's role when sampled responses contain diverse answers, we additionally conduct experiments with N=3N=3, comparing our complete framework with the frozen backbone and VAU-R1 on three datasets. Each method column reports QA accuracy (%) under w/o Think and w/ Think, respectively. The results are reported in the table below.

DatasetFrozen backbone (w/o / w/ Think)VAU-R1 (w/o / w/ Think)Ours, N=3 (w/o / w/ Think)
MSAD85.83 / 82.5088.33 / 87.0892.50 / 90.00
UCF-Crime91.63 / 83.2792.03 / 91.6392.83 / 92.43
ECVA85.58 / 75.8189.53 / 86.5191.55 / 89.95

As shown in the table, our method improves QA accuracy over the frozen backbone by 1.20–14.14 percentage points across all six settings. Furthermore, compared with VAU-R1, our method achieves improvements of 0.80–4.17 percentage points. Therefore, the complete framework remains effective when retained groups can contain competing answers.

Q2. What is svas_{\mathrm{va}}? Does the anchor improve over simple self-training with the same samples, generated text, and KL regularization?

(1) svas_{\mathrm{va}} is an advantage-scaling coefficient applied only to the virtual-anchor branch, because svas_{\mathrm{va}} separately controls the policy-gradient strength contributed by unanimous groups and can be reduced to limit overly aggressive parameter updates during online adaptation. We set sva=1s_{\mathrm{va}}=1 in all experiments, and we will explicitly clarify the definition and value in the revised manuscript.

(2) To explain the difference between simple self-training with KL regularization and our method, we consider unanimous and non-unanimous groups.

  • For unanimous groups, the virtual anchor assigns the same positive advantage to the retained responses. Consequently, the policy-gradient term locally takes the form of a positively scaled token-level SFT gradient, subject to the same token normalization and inactive clipping. The optimization therefore reinforces the complete generated responses, including their reasoning and answer tokens.

  • For non-unanimous groups, however, direct SFT increases the likelihood of every training response, whereas group-relative advantages allow the policy objective to reinforce higher-reward responses and suppress lower-reward responses. Therefore, the key distinction is that advantage-weighted optimization can use answer disagreement to provide a discriminative learning signal, rather than positively fitting all retained responses.

(3) To evaluate whether directly fitting the same generated responses can achieve comparable performance, we conduct a token-level SFT + KL control on MSAD with N=3N=3. Both methods use the same filtered samples and complete generated responses, including reasoning tokens where present, and retain KL regularization. For the self-training control, we replace the advantage-weighted policy objective with token-level negative log-likelihood over the complete responses. To assess whether the difference in optimization translates into better task performance, we compare QA accuracy under w/o Think and w/ Think. In the table, Filtered full-response SFT + KL denotes the direct self-training control, and Ours denotes our complete adaptation method. The results are reported below.

Methodw/o Thinkw/ Think
Filtered full-response SFT + KL90.8387.92
Ours92.5090.00

As shown in the table, our method improves accuracy over direct self-training by 1.67 and 2.08 percentage points under w/o Think and w/ Think, respectively. Therefore, even with matched generated supervision and KL regularization, directly fitting the responses does not achieve the same accuracy as our adaptation method.

Furthermore, we observe degeneration of explicit reasoning output in the w/ Think SFT + KL run. Among 240 predictions, only 111 contain a <think> reasoning section. Starting at step 53, the remaining 129 predictions contain only <answer>X</answer>, despite the instruction to generate reasoning. Thus, including reasoning tokens in the self-training targets does not by itself ensure that explicit reasoning output is maintained during continual adaptation.

In summary, our method shares a positive self-training interpretation for unanimous groups, but additionally uses relative rewards to distinguish competing responses in non-unanimous groups. The matched SFT + KL comparison supports the effectiveness of the complete advantage-weighted adaptation method over directly fitting the same generated responses.

Q3. How do cross-query voting without updates and basic test-time RL perform under the same generation budget? Can the frozen backbone be evaluated with the current inference implementation?

(1) Yes, we re-evaluate the frozen backbone using the current inference implementation, with the same video preprocessing, prompts, visual-token budget, and final evaluation decoding settings as our method. Therefore, the frozen-backbone results provide a reference under matched inference conditions.

(2) We compare inference-time voting and basic test-time RL as follows.

(a) To distinguish the gains from inference-time voting from those of parameter adaptation, we evaluate a frozen-model dual-query voting baseline on MSAD. For each sample, we generate three responses to the original query and three to the reformulated query, then pool all six responses to obtain a majority-vote prediction. Therefore, the voting baseline matches our N=3N=3 configuration in query views and rollout count, while performing no parameter updates.

We compare the frozen backbone, the frozen backbone with dual-query voting, and our complete method. The two accuracy columns report QA accuracy (%) under w/o Think and w/ Think, respectively. The results are reported in the table below.

Methodw/o Thinkw/ Think
Frozen backbone85.8382.50
Frozen backbone + dual-query voting87.0885.00
Ours92.5090.00

As shown in the table, dual-query voting improves accuracy over the frozen backbone by 1.25 and 2.50 percentage points under w/o Think and w/ Think, respectively. Furthermore, our method improves over dual-query voting by 5.42 and 5.00 percentage points. Therefore, the performance gains cannot be explained solely by additional query views and rollout aggregation; parameter adaptation provides further benefits.

(b) To further assess basic test-time RL under a matched generation budget, Table 3 of the manuscript compares binary 0/1 reward adaptation with entropy-aware reward adaptation. As shown in Table 3, binary-reward adaptation improves accuracy over the frozen backbone in five of the six settings, with gains of 0.49–9.35 percentage points, and matches the backbone on UCF-Crime under w/o Think. Furthermore, entropy-aware reward adaptation improves over binary-reward adaptation in all six settings by 0.40–2.51 percentage points. Therefore, basic test-time RL already provides useful adaptation, while incorporating entropy into reward construction yields additional improvements under the matched generation budget.

Reviewer_avZeOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper formulates anomalous video understanding as an online test-time reinforcement learning problem. The authors use Qwen2.5-VL-3B-Instruct as the backbone model, formulate AVU as a multiple-choice question answering task over an unlabeled test stream, and continuously update the model parameters during testing to adapt to changing video distributions. To address the issue that test-time reinforcement learning relies on self-generated pseudo-supervision, which may lead to unreliable updates, the authors propose improvements at three levels. At the sample level, Dual-Query Consistency Filtering compares the prediction consistency between the original question and its semantically equivalent reformulation to filter unreliable samples. At the generation level, Entropy-Aware Consensus Reward combines answer consensus with token-level generation uncertainty. At the optimization level, Virtual Negative Anchor provides a relative reward signal for unanimous rollout groups. The model is then updated at test time using GRPO. Experiments are conducted on the three subsets of VAU-Bench, namely MSAD, UCF-Crime, and ECVA, under both w/o Think and w/ Think settings. The results show that the proposed method outperforms the frozen Qwen2.5-VL-3B across all six QA settings and further demonstrates cross-task transfer from QA test-time adaptation to anomaly classification and fine-grained anomaly recognition. In particular, under the w/ Think setting on ECVA, the accuracy increases from 75.81% to 90.18%.

Critical strengths
  1. The problem formulation is clear, and the method design has a good correspondence between the identified problems and the proposed solutions. The paper categorizes reliability issues in test-time reinforcement learning into three levels: sample, generation, and optimization, and proposes Dual-Query Consistency Filtering, Entropy-Aware Consensus Reward, and Virtual Negative Anchor, respectively. Overall, the method is logically well organized.
  2. The experimental evaluation is relatively comprehensive. The authors conduct experiments on MSAD, UCF-Crime, and ECVA under multiple settings, and further validate the effectiveness of the proposed method through ablation and cross-task experiments. In particular, under the w/ Think setting on ECVA, the accuracy increases from 75.81% to 90.18%, demonstrating a substantial performance improvement.
Critical weaknesses
  1. Dual-Query Consistency and token-level entropy are both indirect proxies for pseudo-label reliability, and there is insufficient theoretical or empirical evidence to establish that they are consistently correlated with prediction correctness. A model may stably produce the same but incorrect answer for two semantically equivalent queries, or generate an incorrect prediction with low entropy and high confidence. Therefore, relying solely on consistency and entropy for sample filtering and reward construction may not be sufficient to effectively prevent erroneous supervision signals.
  2. The actual contribution of the entropy-aware reward remains insufficiently characterized. Although the proposed reward can distinguish rollouts with different confidence in non-unanimous groups, it provides identical rewards when all retained rollouts select the same answer, in which case the optimization signal is mainly recovered by the virtual negative anchor. The paper does not report how frequently retained samples are unanimous versus non-unanimous after dual-query filtering, making it difficult to assess how often the entropy-aware component provides meaningful reward differentiation in practice.
Questions for authors
  1. The paper uses Dual-Query Consistency and token-level entropy as important indicators of pseudo-label reliability. How do the authors demonstrate that these two indicators are indeed reflective of prediction correctness? For example, could the original and reformulated queries produce the same but incorrect answer, or could the model assign low entropy and high confidence to an incorrect answer?
  2. After Dual-Query Consistency Filtering, what proportions of the retained rollout groups are unanimous and non-unanimous, respectively? How does this distribution affect the practical contribution of the Entropy-Aware Consensus Reward?
  3. During online test-time adaptation, does the parameter update produced from one test sample affect the predictions and adaptation of subsequent test samples? If so, is the model performance sensitive to the order of test samples?
  4. Compared with existing Test-Time RL methods, the proposed method requires additional query reformulation, multiple rollouts, and GRPO updates. How much additional computational and inference overhead does this introduce?
4 comments

We sincerely thank the reviewer for the careful evaluation and constructive questions, which have helped us clarify the method and strengthen its empirical validation.

Q1. The paper uses Dual-Query Consistency and token-level entropy as important indicators of pseudo-label reliability. How do the authors demonstrate that these two indicators are indeed reflective of prediction correctness? For example, could the original and reformulated queries produce the same but incorrect answer, or could the model assign low entropy and high confidence to an incorrect answer?

During test-time adaptation, ground-truth labels are unavailable, and supervision is therefore derived from self-generated pseudo-labels. However, self-generated pseudo-labels inevitably contain errors. So, our design aims to maximize pseudo-label accuracy to provide reliable supervision for effective model adaptation. To answer the question, we additionally perform two experiments as follows:

(1) To evaluate the quality of pseudo-labels, we conduct experiments on three subsets of VAU-bench (ECVA, MSAD, and UCF-Crime), using the frozen Qwen2.5-VL-3B-Instruct model with three independent responses sampled per query.

We compare three evaluation protocols:

  • (a) Single-sample accuracy (no voting): The percentage of correct answers over all test samples, using one response to the original question per sample without voting.
  • (b) Original-majority vote accuracy: The percentage of correct majority answers among samples for which at least half of the responses to the original question agree on the same answer.
  • (c) Ours (Dual-query-retained accuracy): The percentage of correct agreed majority answers among samples for which both the original and reformulated questions have a majority answer and the two majority answers match.

The results are reported in the table below.

DatasetSettingSingle-sample accuracy (no voting)Original-majority vote accuracyOurs (Dual-query-retained accuracy)
ECVAw/o Think85.58%89.73%92.82%
ECVAw/ Think75.81%87.91%91.57%
MSADw/o Think85.83%90.76%93.33%
MSADw/ Think82.50%90.52%94.44%
UCF-Crimew/o Think91.63%90.34%93.42%
UCF-Crimew/ Think83.27%88.26%90.48%

As shown in the table, dual-query-retained accuracy exceeds single-sample accuracy (no voting) by 1.79–15.76 percentage points across all six settings, indicating that the pseudo-labels retained after vote aggregation and consistency filtering are more accurate than unfiltered single-response predictions. Compared with original-majority vote accuracy, dual-query-retained accuracy further increases by 2.22–3.92 percentage points, demonstrating the additional benefit of cross-query consistency filtering beyond original-query majority voting. Together, the comparisons show that the retained pseudo-labels provide more reliable pseudo-supervision for test-time adaptation.

(2) To evaluate whether lower token entropy is associated with higher prediction correctness, we analyze generated responses from samples retained by dual-query consistency filtering under the w/o Think setting on ECVA, MSAD, and UCF-Crime. For each generated response, we compute the mean entropy over all generated tokens and label the response as correct or incorrect by comparing the extracted answer with the ground-truth answer. We then calculate ROC-AUC using negative mean token entropy as the score and response correctness as the binary label. The results are reported in the table below.

DatasetEntropy–correctness ROC-AUC, w/o Think
ECVA0.891
MSAD0.873
UCF-Crime0.881

As shown in the table, the ROC-AUC values range from 0.873 to 0.891 across the three datasets, which shows that, among responses from retained samples under w/o Think, correct responses tend to have lower mean token entropy than incorrect responses. Therefore, mean token entropy helps distinguish correct from incorrect responses and provides an empirical basis for assessing pseudo-label reliability.

In summary, although agreement between the original and reformulated queries does not guarantee correctness and low token entropy cannot completely rule out incorrect answers, the experiments show that dual-query consistency filtering improves retained pseudo-label accuracy and that lower token entropy correlates with higher response correctness under w/o Think. Therefore, dual-query consistency and token-level entropy provide empirically supported criteria for assessing pseudo-label reliability and obtaining more reliable supervision for test-time adaptation.

Q2. After Dual-Query Consistency Filtering, what proportions of the retained rollout groups are unanimous and non-unanimous, respectively? How does this distribution affect the practical contribution of the Entropy-Aware Consensus Reward?

Under the default N=2N=2, strict-majority filtering retains only unanimous groups because both responses within each query view must agree and the majority answers across query views must match. Therefore, 100% of retained groups are unanimous, and all retained rollouts receive the same answer-level reward. Nevertheless, we consider equal rewards appropriate because, from the perspective of answer-level supervision, responses supporting the same cross-query-consistent answer share the same supervision target. Besides, the experiments in our response to Q1 demonstrate that dual-query consistency filtering improves pseudo-label accuracy. Therefore, responses supporting the same retained answer need not receive different answer-level rewards solely because their generation confidence differs.

Beyond unanimous groups, confidence-based weighting becomes useful when retained groups contain competing answers, because generation confidence can further inform reward allocation across answers. To examine how frequently competing answers remain after filtering, we conduct additional experiments with batch size 2 and N=3N=3. For each retained original-query rollout group, we classify the group as unanimous if all three responses select the same answer and non-unanimous otherwise. In the table, Retained denotes the total number of retained groups, while Unanimous and Non-unanimous report the counts and percentages relative to that total. The results are reported in the table below.

DatasetSettingRetainedUnanimousNon-unanimous
ECVAw/ Think415357 (86.02%)58 (13.98%)
MSADw/ Think216184 (85.19%)32 (14.81%)
UCF-Crimew/ Think210178 (84.76%)32 (15.24%)

As shown in the table, unanimous groups account for 84.76%–86.02% of retained groups, so enabling learning from equal-reward groups is essential for using most of the retained supervision. Meanwhile, 13.98%–15.24% of retained groups contain competing answers, allowing entropy-based weighting to influence reward allocation across answers. Accordingly, the two components address different group structures:

  • For unanimous groups, agreement across rollouts and query views provides consensus supervision. However, standard GRPO assigns zero group-relative advantages to identical rewards, preventing consensus supervision from producing a policy-gradient signal. To restore the learning signal, the virtual negative anchor introduces a lower-reward reference into advantage normalization, thereby assigning positive advantages to the real consensus rollouts and enabling unanimous groups to guide model learning.
  • For non-unanimous groups, the Entropy-Aware Consensus Reward converts generation uncertainty into confidence weights, aggregates the weights by answer, and normalizes the aggregated weights into answer-level rewards. Therefore, when competing answers remain, reward allocation incorporates generation confidence in addition to answer frequency.

In summary, the Entropy-Aware Consensus Reward and the virtual negative anchor play different, complementary roles: entropy-based weighting incorporates confidence into reward allocation among competing answers, while the virtual negative anchor restores the learning signal when unanimous answers yield identical rewards. Together, the two components enable learning from both non-unanimous and unanimous groups retained by dual-query consistency filtering.

Q3. During online test-time adaptation, does the parameter update produced from one test sample affect the predictions and adaptation of subsequent test samples? If so, is the model performance sensitive to the order of test samples?

(1) Yes, parameter updates affect subsequent predictions and adaptation, because the updated parameters are retained after each batch and used to process subsequent batches. Each dataset is independently initialized from the original backbone, so parameter updates accumulate only within the test stream of the same dataset.

(2) To evaluate sensitivity to test-sample order, we conduct three runs using seeds 42, 43, and 44 to shuffle the test samples. For each run, we initialize the model from the original backbone and perform the complete online adaptation and evaluation procedure. We report the mean and standard deviation of QA accuracy across the three orders under w/o Think and w/ Think. The results are shown in the table below as mean ± standard deviation (%).

Datasetw/o Thinkw/ Think
MSAD91.94 ± 0.6488.47 ± 0.48
UCF-Crime93.63 ± 0.6991.63 ± 0.40
ECVA92.16 ± 0.2790.18 ± 0.23

As shown in the table, the standard deviations range from 0.23 to 0.69 percentage points across all three datasets and both reasoning settings, indicating limited variation in overall QA accuracy across the shuffled test orders. Therefore, although accumulated parameter updates affect subsequent samples, overall performance remains stable across the evaluated orders.

In summary, our method retains parameter updates to support continual adaptation, while the small accuracy variations across shuffled runs demonstrate low sensitivity to test-sample order under the evaluated settings.

Q4. Compared with existing Test-Time RL methods, the proposed method requires additional query reformulation, multiple rollouts, and GRPO updates. How much additional computational and inference overhead does this introduce?

Multiple rollouts and GRPO updates are shared with existing GRPO-based test-time RL methods. Therefore, the additional computation introduced by our method mainly consists of video description generation, query reformulation, and sampling responses to the reformulated query for consistency filtering. Description generation and query reformulation use frozen models, so their outputs can be cached to avoid repeated computation. Furthermore, the virtual negative anchor only adds a scalar reference to advantage normalization and does not require generating an additional response.

To quantify the computational cost, we measure the complete pipeline on UCF-Crime with 251 videos, using two NVIDIA A800 GPUs, a batch size of 2, and N=3N=3 rollouts per query view. The measured runtime includes video description generation, query reformulation, rollout sampling, parameter updates, test inference, and evaluation. For w/ Think, the maximum response length is 512 tokens, while the maximum response length is 20 tokens for w/o Think. For comparison, we evaluate a basic RL baseline under the same hardware, batch size, and response-length settings. The baseline samples three responses to the original question, computes rewards, updates model parameters, and performs test inference and evaluation. We report average processing time per video and peak memory usage per GPU. The results are reported in the table below.

SettingBasic RL: average time per videoOurs: average time per videoBasic RL: peak memory per GPUOurs: peak memory per GPU
w/o Think14.27 s21.43 s≈25 GB≈25 GB
w/ Think42.15 s68.57 s≈25 GB≈25 GB

As shown in the table, our method increases average processing time relative to the basic RL baseline by 7.16 seconds per video under w/o Think and 26.42 seconds per video under w/ Think, corresponding to approximately 1.50× and 1.63× the baseline cost, respectively. The additional time reflects video description generation, query reformulation, and rollout sampling for the reformulated query. Furthermore, the longer responses generated under w/ Think increase processing time for both methods. However, peak memory usage remains approximately 25 GB per GPU across both methods and settings. Therefore, the additional computation primarily increases time cost without increasing the measured peak memory requirement. For deployment scenarios that permit moderate processing delays, such as periodic surveillance analysis, we consider the additional overhead acceptable.

AuthorsOpenReview ↗

Looking forward to your feedback

Dear Reviewer avZe ,

We sincerely appreciate your time and effort in reviewing this work. We understand that your schedule may be quite busy. As the author-reviewer discussion phase draws to a close, we kindly request your attention to our responses. We would be grateful to know whether our replies have effectively addressed your concerns, and whether there are any additional questions or points you would like to discuss. Should you find our answers satisfactory, we also hope you may consider adjusting your score accordingly. We look forward to the opportunity to discuss this further with you. Thank you for your thoughtful consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.