- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes SeTeR, a framework for selectively refining temporal grounding predictions. Starting from the output of a frozen TimeLens2-8B model, it generates local revisions and uses a learned verifier to estimate whether each revision would improve or degrade the current prediction. A reinforcement learning policy then decides which edits to apply and when to stop.
The method improves mIoU by 1.22, 1.58, and 1.35 points on three TimeLens benchmarks. The authors also report improvements on OMTG under cross-dataset transfer.
Critical strengths
-
The motivation is clear and reasonable. A candidate interval can look plausible without being better than the current prediction. Evaluating the expected change in grounding quality, rather than candidate quality alone, is a useful idea. The ungated selection results in Table 3 help demonstrate this problem.
-
The experimental evaluation is fairly thorough. The method shows consistent improvements across three benchmarks. The authors also include ablations, oracle results, runtime measurements, and analyses of successful and failed revisions, which help explain the behavior of the method.
Critical weaknesses
-
The benefit of reinforcement learning is not convincing enough. In Table 5, the full method reaches 71.55 mIoU on QVHighlights, compared with 71.35 using greedy selection and 71.25 using direct gain regression. The RL policy therefore contributes only 0.20 mIoU over the greedy strategy. Since learning edit utility already accounts for most of the improvement, it is unclear whether reinforcement learning is necessary. A behavior-cloning-only baseline is also missing, and the reported differences are not supported by multi-seed results.
-
The risk-aware mechanism does not consistently protect accurate predictions. Table C.3 shows that, among revised predictions with initial IoU >= 0.9, 55.3%, 73.2%, and 63.7% are degraded across the three datasets. These numbers do not necessarily indicate a negative overall gain, but they are concerning given the paper's emphasis on avoiding harmful revisions. The authors should report the average improvement and degradation magnitudes for these cases, rather than only the number of affected queries.
Questions for authors
-
How much of the improvement comes specifically from REINFORCE? Could the authors report a behavior-cloning-only baseline and results across multiple random seeds?
-
Why are so many high-quality initial predictions degraded after revision? What is the average net IoU change for these predictions?
-
For the OMTG experiments, how much of the improvement comes from split edits? A result without split edits would help clarify whether the method actually improves event separation or mainly benefits from boundary refinement.
4 comments
High-IoU Revision Reliability and OMTG Split Ablation
We sincerely thank the reviewer for the thoughtful questions regarding high-quality prediction reliability and the contribution of split edits. These suggestions motivated two additional analyses that clarify how selective refinement affects accurate predictions and multi-event grounding.
Q2 & W2. Why are initially high-quality predictions sometimes degraded?
The degradation percentages in Table C.3 are conditioned on predictions that both have initial IoU ≥ 0.9 and undergo revision. They should not be interpreted as degradation rates among all initially accurate predictions.
Following the reviewer's suggestion, we computed the average magnitudes of positive and negative changes, together with the net IoU effect.
Additional high-IoU analysis (IoU in [0, 1]):
| Dataset | Initial IoU ≥ 0.9 | RL Revised | Mean Gain | Mean Loss | Net ΔIoU (Revised) |
|---|---|---|---|---|---|
| QVHighlights | 690 | 161 | 0.075 | 0.030 | +0.010 |
| Charades | 748 | 123 | 0.068 | 0.046 | −0.016 |
| ActivityNet | 1,276 | 355 | 0.080 | 0.025 | +0.009 |
Mean Gain and Mean Loss are averaged over improved and degraded revised predictions, respectively. Net ΔIoU includes revised predictions whose IoU remains unchanged.
These results clarify why degradation frequency alone does not fully characterize reliability. Although Table C.3 reports more degraded than improved predictions among revised high-IoU samples, their average improvement magnitudes exceed their average degradation magnitudes. Consequently, the revised subsets achieve positive net IoU changes on QVHighlights and ActivityNet, while Charades retains a negative net change.
More importantly, we additionally evaluate the impact on all initially high-IoU predictions, assigning zero IoU change to predictions that were not revised. This captures the combined effects of revision frequency and revision quality.
Population-level comparison with greedy selection:
| Dataset | RL Revision Rate | Greedy Revision Rate | RL Net ΔIoU ↑ | Greedy Net ΔIoU ↑ |
|---|---|---|---|---|
| QVHighlights | 23.33% | 38.12% | +0.0023 | −0.0126 |
| Charades | 16.44% | 13.90% | −0.0026 | −0.0083 |
| ActivityNet | 27.82% | 44.20% | +0.0024 | −0.0319 |
Both methods are evaluated on the same initial high-IoU populations. Greedy revises 263, 104, and 564 predictions on QVHighlights, Charades, and ActivityNet, respectively.
Full RL achieves better population-level net IoU changes than greedy on all three datasets. On QVHighlights and ActivityNet, the full policy achieves positive net changes, whereas greedy produces negative changes. On Charades, the full policy revises slightly more high-IoU predictions but substantially reduces the overall net degradation.
These results show that the risk-aware mechanism mitigates damage through the combined effects of edit selection and revision quality. Its objective is to reduce harmful revisions rather than guarantee that every accepted edit is beneficial.
Q3. How much of the OMTG improvement comes from split edits?
We appreciate the suggestion to distinguish event separation from boundary refinement. We therefore evaluated SeTeR with split edits disabled, preserving the original QVHighlights-to-OMTG transfer setting without any OMTG training.
Additional OMTG ablation:
| Metric | TimeLens2 | SeTeR w/o Split | Full SeTeR |
|---|---|---|---|
| C-Acc ↑ | 16.88 | 16.88 | 25.00 |
| [email protected] ↑ | 49.04 | 48.95 | 58.26 |
| [email protected] ↑ | 39.30 | 39.21 | 49.42 |
| [email protected] ↑ | 26.74 | 26.85 | 35.94 |
| tIoU ↑ | 48.93 | 49.00 | 51.56 |
| EtF1 ↑ | 13.21 | 13.24 | 19.06 |
Without split edits, the number of predicted intervals remains unchanged, so C-Acc stays at the baseline value of 16.88. The remaining boundary-only modifications yield results close to the original TimeLens2 predictions.
By contrast, restoring split edits increases C-Acc to 25.00, [email protected] to 49.42, and EtF1 to 19.06. Relative to the no-split variant, these correspond to improvements of 8.12, 10.21, and 5.82 points, respectively.
The results demonstrate that the OMTG improvements are primarily associated with the ability to revise event structure, rather than merely adjusting the boundaries of an unchanged number of intervals. In particular, split edits enable the model to separate occurrences that were initially merged into a single interval.
This directly supports our central motivation for refining the complete interval set, including both temporal boundaries and event structure.
We sincerely appreciate the reviewer's constructive suggestions, which have strengthened the empirical analysis of our framework.
Kind Reminder
Dear Reviewer,
We hope you are having a good week. Thank you again for your insightful comments. To address your questions, we have added controlled evaluations of REINFORCE, analyzed net IoU changes for initially accurate predictions, and conducted an OMTG ablation to isolate the contribution of split edits.
As the discussion deadline draws closer, we would greatly appreciate hearing whether these analyses address your concerns. Please let us know if further details or experiments would be useful; we would be happy to follow up while time remains for discussion.
If you find that the additional results have addressed your concerns, would you kindly consider raising your rating for our paper?
Thank you very much for your time and consideration. We look forward to your feedback.
Final Revision: RL Controls, Revision Reliability, and Split Edits
Thank you for your suggestions. We have updated the final version of the paper to include controlled RL comparisons, additional high-IoU reliability analyses, and a no-split ablation.
- Contribution of REINFORCE. Over three independent end-to-end seeds on QVHighlights, including verifier retraining, full RL achieves 71.52±0.07 mIoU, compared with 71.31±0.12 for greedy, 71.08±0.42 for BC-only, and 71.34±0.29 for budget-matched BC.
- Initially accurate predictions. We report conditional improvement and degradation magnitudes, together with net changes over all initially high-IoU predictions. On QVHighlights, Charades, and ActivityNet, RL produces population-level net IoU changes of +0.0023, −0.0026, and +0.0024, compared with −0.0126, −0.0083, and −0.0319 for greedy, respectively. These changes use the [0,1] IoU scale.
- Contribution of split edits. On OMTG, disabling splits yields 16.88 C-Acc, 39.21 [email protected], and 13.24 EtF1. Full SeTeR achieves 25.00, 49.42, and 19.06, respectively, supporting the contribution of event separation.
REINFORCE Contribution: BC-Only and Multi-Seed Comparisons
We sincerely thank the reviewer for the careful assessment and for recognizing the motivation and experimental coverage of our work. We particularly appreciate the constructive suggestion to isolate the contribution of REINFORCE through behavior-cloning baselines and multi-seed evaluation.
Q1 & W1. How much improvement is specifically attributable to REINFORCE?
To isolate the contribution of policy-gradient optimization, we conducted two additional experiments.
First, we introduced BC-only, retaining the original behavior-cloning initialization while removing REINFORCE. This evaluates the effectiveness of supervised policy imitation alone.
Second, we introduced BC-only (Matched) to control for the additional training received by the full RL policy. We extended supervised BC training to match the overall policy-training budget of the full method, allowing us to distinguish the effect of the RL objective from simply performing more training.
We also extended the comparison with greedy selection to all three datasets.
Additional comparison (mIoU ↑):
QVHighlights reports mean±standard deviation over three independent end-to-end seeds, including verifier retraining; Charades and ActivityNet report single runs. The original Table 5 values (71.35 greedy; 71.55 full) correspond to one run.
Additional BC training improves performance but does not eliminate the RL advantage. Compared with the original BC-only policy, matched BC improves mIoU by 0.20, 0.24, and 0.26 points across the three datasets. Full RL nevertheless achieves further improvements of 0.17, 0.31, and 0.18 points, respectively.
Full RL also outperforms greedy selection by 0.41, 0.53, and 0.21 mIoU points. On QVHighlights, the full policy shows lower run-to-run variation (±0.07) than greedy (±0.12) and matched BC (±0.29).
These results provide evidence of an incremental contribution from REINFORCE beyond both verifier-guided greedy selection and supervised training.
The distinction is also reflected in the learning objectives. BC imitates reference-guided edit choices at individual states, whereas REINFORCE optimizes trajectory-level rewards incorporating realized utility changes, penalties for harmful and unnecessary edits, and final prediction quality. This directly supports learning sequential editing and stopping decisions.
We appreciate the reviewer's suggestion, which has enabled a more controlled assessment of the policy-gradient contribution.