- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces a reliability-aware test-time reinforcement learning (TTRL) framework for anomalous video understanding (AVU) with Video-LLMs. It addresses three limitations of existing TTRL methods: inconsistent pseudo-labels for semantically equivalent queries, reward estimation that overlooks generation uncertainty, and vanishing GRPO advantages when all rollouts agree. To address these issues, the framework combines dual-query consistency filtering, an entropy-aware consensus reward, and a virtual negative anchor for advantage normalization. Experiments on VAU-Bench with Qwen2.5-VL-3B show consistent improvements over frozen and supervised baselines in multiple-choice question answering. The method also improves anomaly classification performance without additional task-specific supervision.
Critical strengths
1.The paper addresses several practical limitations of test-time reinforcement learning through three complementary designs. The method improves pseudo-label reliability through dual-query consistency filtering, provides finer-grained rewards by incorporating prediction uncertainty, and mitigates GRPO advantage collapse without additional rollouts.
2.The experiments are comprehensive, covering multiple standard distribution-shift benchmarks, comparisons with several baselines, and ablation studies to assess the contribution of each component.
Critical weaknesses
- The core idea of using reliability or confidence measures for sample selection in test-time adaptation (TTA) is not new. EATA[1] explicitly selects reliable samples based on prediction entropy and filters redundant samples using prediction similarity. Similarly, SAR[2] employs entropy-based reliable sample filtering to mitigate unstable model updates. However, the paper does not clearly distinguish its proposed reliability metric from these existing approaches or sufficiently justify its novelty.
- The motivation is not sufficiently clear. While the Introduction discusses the advantages of test-time adaptation (TTA) for anomalous video understanding, it does not adequately explain why test-time reinforcement learning (TTRL) is particularly suitable for this task or what limitations of existing TTA methods motivate its use.
[1] Niu, S., Wu, J., Zhang, Y., Chen, Y., Zheng, S., Zhao, P., and Tan, M. Efficient Test-Time Model Adaptation without Forgetting. Proceedings of the 39th International Conference on Machine Learning (ICML), pp. 16888–16905, 2022. [2] Niu, S., Wu, J., Zhang, Y., Wen, Z., Chen, Y., Zhao, P., and Tan, M. Towards Stable Test-Time Adaptation in Dynamic Wild World. International Conference on Learning Representations (ICLR), 2023.
Questions for authors
- How are the thresholds for distinguishing reliable and unreliable samples determined? How sensitive is the proposed method to different threshold values?
- How does the proposed method compare with more recent TTA/TTRL approaches?
- Did you experiment with simpler pseudo-label self-training using cross-entropy on the majority label (without GRPO)? A comparison would help isolate the value of group-normalized policy-gradient updates versus supervised pseudo-labeling.
5 comments
Q1. How are the thresholds for distinguishing reliable and unreliable samples determined? How sensitive is the proposed method to different threshold values?
(1) We select reliable samples through strict-majority agreement across two query views, rather than an entropy threshold. Specifically, we set the answer-support threshold to 0.5, requiring an answer to receive more than half of the responses in each query view. The majority answers from the original and reformulated queries must also match for the sample to be retained. Because a strict majority gives the selected answer more support than all alternatives combined, while cross-query agreement checks whether the answer remains stable under semantically equivalent reformulation, we combine both conditions to select reliable pseudo-supervision. The same rule is applied across all datasets without dataset-specific threshold tuning. The parameters and are used for reward redistribution, not for determining whether a sample is accepted.
(2) To examine sensitivity to different rollout counts and their corresponding majority requirements, we compare on MSAD under w/o Think, with batch size fixed at 2. The minimum required votes per query view are 2, 2, and 3, respectively, corresponding to agreement levels of 100%, 66.67%, and 75%. All configurations additionally require the majority answers across the two query views to match. The table reports the rollout count, required votes, minimum agreement level, and resulting QA accuracy. The results are shown below.
| Rollouts per view | Required votes per view | Minimum agreement | QA accuracy |
|---|---|---|---|
| 2 | 100% | 92.50% | |
| 2 | 66.67% | 92.50% | |
| 3 | 75% | 91.67% |
As shown in the table, compared with the frozen backbone’s accuracy of 85.83%, the three configurations improve accuracy by 6.67, 6.67, and 5.84 percentage points, respectively. Furthermore, and achieve identical accuracy, while is only 0.83 percentage points lower. Therefore, the method consistently outperforms the frozen backbone and exhibits limited performance variation across the evaluated rollout counts and corresponding majority requirements.
In summary, sample selection uses a fixed cross-query strict-majority rule rather than a dataset-specific entropy threshold. The results demonstrate stable performance across the evaluated sampling and consensus configurations.
Q2. How does the proposed method compare with more recent TTA/TTRL approaches?
To evaluate our method against recent test-time reinforcement learning approaches, we have already included TTRL (NeurIPS 2025) [3] in Table 3, corresponding to the Binary 0/1 reward baseline. We additionally compare with TTRV (CVPR 2026) [4] in the supplementary experiments.
To maintain consistent experimental settings, all three methods use Qwen2.5-VL-3B-Instruct, three sampled responses per query view, and a batch size of 2. For each batch, all methods perform one parameter update using the unlabeled inputs and then immediately evaluate the updated model on the same batch. Updated parameters are retained for subsequent batches. All methods follow the same evaluation protocol, including video preprocessing, prompts, visual-token budget, and final evaluation decoding settings.
We evaluate all methods on UCF-Crime, MSAD, and ECVA under both w/o Think and w/ Think. Each row corresponds to one method, and the columns report QA accuracy (%) for each dataset and reasoning setting. The results are presented in the table below.
| Method | UCF-Crime w/o Think | UCF-Crime w/ Think | MSAD w/o Think | MSAD w/ Think | ECVA w/o Think | ECVA w/ Think |
|---|---|---|---|---|---|---|
| TTRL | 91.63 | 86.06 | 87.08 | 85.42 | 86.07 | 85.16 |
| TTRV | 92.03 | 90.84 | 87.92 | 87.50 | 89.27 | 89.04 |
| Ours | 92.83 | 92.43 | 92.50 | 90.00 | 91.55 | 89.95 |
As shown in the table, compared with TTRL, our method improves accuracy in all six settings by 1.20–6.37 percentage points, with an average improvement of 4.64 percentage points. Furthermore, compared with TTRV, our method achieves improvements of 0.80–4.58 percentage points, averaging 2.11 percentage points across the six settings. Therefore, the performance gains hold across all three datasets and both reasoning settings.
In summary, our method consistently outperforms TTRL and TTRV with the same backbone, per-view rollout count, batch size, and evaluation protocol, demonstrating its effectiveness relative to recent test-time reinforcement learning approaches.
[3] TTRL: Test-Time Reinforcement Learning. NeurIPS, 2025.
[4] TTRV: Test-Time Reinforcement Learning for Vision Language Models. CVPR, 2026.
Q3. Did you experiment with simpler pseudo-label self-training using cross-entropy on the majority label (without GRPO)? A comparison would help isolate the value of group-normalized policy-gradient updates versus supervised pseudo-labeling.
(1) To explain the distinction between the optimization objectives, we consider the composition of retained rollout groups. For unanimous groups, equal positive advantages give the policy update a positive self-training interpretation. However, when retained groups contain competing answers, full-response SFT increases the likelihood of every training response, whereas group-relative advantages enable the policy objective to reinforce higher-reward responses and suppress lower-reward responses. Therefore, advantage-weighted optimization can use answer disagreement to provide a discriminative learning signal rather than positively fitting all retained responses.
(2) Following the reviewer’s suggestion, we conduct an additional self-training experiment on MSAD with to evaluate the value of advantage-weighted policy optimization over direct pseudo-label self-training. We also retain KL regularization in both methods to constrain deviation from the reference policy and limit excessive model updates. For the self-training control, we replace the advantage-weighted policy objective with token-level cross-entropy over the complete generated responses, including reasoning tokens where present, rather than supervising only the majority-answer label.
In the table, Full-response SFT + KL denotes the direct self-training control, and Ours denotes our adaptation method. The two columns report QA accuracy (%) under w/o Think and w/ Think, respectively. The results are reported below.
| Method | w/o Think | w/ Think |
|---|---|---|
| Full-response SFT + KL | 90.83 | 87.92 |
| Ours | 92.50 | 90.00 |
As shown in the table, our method improves accuracy over full-response SFT + KL by 1.67 and 2.08 percentage points under w/o Think and w/ Think, respectively. Therefore, even with the same generated supervision and KL regularization, directly fitting the responses does not achieve the accuracy of our adaptation method.
Furthermore, we observe degeneration of explicit reasoning output in the w/ Think SFT + KL run. Among 240 predictions, only 111 contain a <think> reasoning section. Starting at step 53, the remaining 129 predictions contain only <answer>X</answer>, despite the instruction to generate reasoning. Because the self-training targets include the reasoning tokens present in the generated responses, the degeneration cannot be attributed simply to answer-only supervision. Thus, fitting self-generated texts containing reasoning does not by itself ensure that explicit reasoning output is maintained during continual adaptation.
In summary, our method outperforms full-response self-training under matched generated supervision and KL regularization. Unlike uniformly reinforcing the training responses, advantage-weighted policy optimization can additionally use answer disagreement to distinguish the update directions of competing responses. The accuracy results support the practical value of the optimization approach over direct pseudo-label fitting.
We sincerely thank the reviewer for the thoughtful feedback and constructive suggestions on the motivation, methodological distinctions, and experimental comparisons.
Weakness 1. The core idea of using reliability or confidence measures for sample selection in test-time adaptation (TTA) is not new. EATA[1] explicitly selects reliable samples based on prediction entropy and filters redundant samples using prediction similarity. Similarly, SAR[2] employs entropy-based reliable sample filtering to mitigate unstable model updates. However, the paper does not clearly distinguish its proposed reliability metric from these existing approaches or sufficiently justify its novelty.
We agree that reliability-based sample selection has been studied in existing TTA methods. Our contribution concerns how reliability is assessed and used in generative anomalous-video understanding. We clarify the differences from EATA and SAR below.
(1) Task formulation. EATA and SAR address discriminative prediction, whereas our method adapts a generative video-language model for anomalous-video question answering. Because generated responses can express the same answer through different wording or reasoning paths, we assess agreement at the answer level across generated responses and question formulations.
(2) Reliability criterion. EATA selects low-entropy predictions and filters redundant samples based on prediction similarity, while SAR uses entropy-based reliable sample filtering. In contrast, our method retains a sample only when the original and semantically equivalent reformulated questions produce the same strict-majority answer. Because low-entropy predictions can still disagree across question formulations, confidence alone does not establish cross-query consistency. Our criterion therefore explicitly checks whether the predicted answer remains stable under question reformulation.
(3) Role in optimization. EATA and SAR select reliable predictions for entropy minimization. In our framework, cross-query consistency instead selects response groups that provide reliable answer-level feedback for reward construction. Furthermore, token entropy is converted into confidence weights and aggregated by answer to construct rewards, which are then used for response-level policy optimization. Therefore, cross-query consistency determines which groups provide supervision, while token entropy contributes to the rewards used to optimize the generative policy.
In summary, our contribution lies in assessing reliability through cross-query answer consistency and using the retained answer-level feedback to construct rewards for test-time adaptation in generative anomalous-video understanding. We will clarify the differences in task formulation, reliability criterion, and optimization objective from EATA and SAR in the revised manuscript.
Weakness 2. The motivation is not sufficiently clear. While the Introduction discusses the advantages of test-time adaptation (TTA) for anomalous video understanding, it does not adequately explain why test-time reinforcement learning (TTRL) is particularly suitable for this task or what limitations of existing TTA methods motivate its use.
Our task uses a video-language model to perform reasoning about anomalous videos. However, neither ground-truth answers nor reference reasoning sequences are available at test time. Therefore, adaptation requires an optimization objective that can use feedback derived from the model’s own responses without relying on annotated target sequences.
EATA and SAR use entropy-minimization-based objectives to increase prediction confidence on selected samples. However, when applied to generation, entropy minimization optimizes how certain the output distribution is, whereas the task requires correct answers and sound reasoning. Greater certainty alone does not establish either answer correctness or reasoning quality. In contrast, TTRL converts response rewards into relative advantages for policy optimization, allowing higher-reward responses to be reinforced and lower-reward responses to be suppressed when sampled answers disagree. Therefore, we adopt TTRL to use answer-level feedback derived from repeated sampling and cross-query consistency to construct rewards and guide response-level optimization.
In summary, TTRL is suitable for the task because it uses self-generated answer-level feedback to optimize complete responses without reference answer or reasoning annotations, while allowing relative rewards to distinguish competing responses. We will make the task-specific motivation explicit in the revised Introduction.