acceptodds
Under review as a conference paper at ICLR 2027

Beyond Optimism: Selective Temporal Reinforcement Learning for Video Grounding

Abstract

Video temporal grounding localizes the temporal boundaries and occurrences of queried events. Existing methods either return a final prediction in a single pass or iteratively correct predictions based on candidate validity and overlap. The former leave residual localization errors uncorrected, while the latter overlook the benefit and degradation risk of changing the current answer. Our diagnostic study across three benchmarks shows that accepting the candidate with the highest verifier score degrades more queries than it improves. To address this issue, we propose SeTeR, a Selective Temporal Reinforcement Learning framework that learns the value of temporal edits and decides when to apply them. SeTeR revisits regions identified by a frozen grounder to propose boundary adjustments and splits of merged occurrences. We further introduce a learnable revision verifier that compares the current and proposed interval sets, jointly predicting improvement probability, degradation risk, and positive gain under supervision from the change in complete answer utility. Conditioned on these signals, a reinforcement learning policy selects edits or stops, optimizing grounding gains with penalties for degradation and unnecessary modifications. Experiments with TimeLens2-8B demonstrate improvements across all twelve reported metrics on three temporal grounding benchmarks, including mIoU gains of 1.22, 1.58, and 1.35 points. Transfer from QVHighlights to OMTG further improves event count accuracy by 8.12 points and F1 at the event level by 5.85 points, demonstrating the effectiveness of SeTeR for both boundary localization and event separation.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Video temporal grounding, Reinforcement learning, Learned verification, Multimodal large language models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_WtiCOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes SeTeR, a framework for selectively refining temporal grounding predictions. Starting from the output of a frozen TimeLens2-8B model, it generates local revisions and uses a learned verifier to estimate whether each revision would improve or degrade the current prediction. A reinforcement learning policy then decides which edits to apply and when to stop.

The method improves mIoU by 1.22, 1.58, and 1.35 points on three TimeLens benchmarks. The authors also report improvements on OMTG under cross-dataset transfer.

Critical strengths
  1. The motivation is clear and reasonable. A candidate interval can look plausible without being better than the current prediction. Evaluating the expected change in grounding quality, rather than candidate quality alone, is a useful idea. The ungated selection results in Table 3 help demonstrate this problem.

  2. The experimental evaluation is fairly thorough. The method shows consistent improvements across three benchmarks. The authors also include ablations, oracle results, runtime measurements, and analyses of successful and failed revisions, which help explain the behavior of the method.

Critical weaknesses
  1. The benefit of reinforcement learning is not convincing enough. In Table 5, the full method reaches 71.55 mIoU on QVHighlights, compared with 71.35 using greedy selection and 71.25 using direct gain regression. The RL policy therefore contributes only 0.20 mIoU over the greedy strategy. Since learning edit utility already accounts for most of the improvement, it is unclear whether reinforcement learning is necessary. A behavior-cloning-only baseline is also missing, and the reported differences are not supported by multi-seed results.

  2. The risk-aware mechanism does not consistently protect accurate predictions. Table C.3 shows that, among revised predictions with initial IoU >= 0.9, 55.3%, 73.2%, and 63.7% are degraded across the three datasets. These numbers do not necessarily indicate a negative overall gain, but they are concerning given the paper's emphasis on avoiding harmful revisions. The authors should report the average improvement and degradation magnitudes for these cases, rather than only the number of affected queries.

Questions for authors
  1. How much of the improvement comes specifically from REINFORCE? Could the authors report a behavior-cloning-only baseline and results across multiple random seeds?

  2. Why are so many high-quality initial predictions degraded after revision? What is the average net IoU change for these predictions?

  3. For the OMTG experiments, how much of the improvement comes from split edits? A result without split edits would help clarify whether the method actually improves event separation or mainly benefits from boundary refinement.

4 comments
AuthorsOpenReview ↗

REINFORCE Contribution: BC-Only and Multi-Seed Comparisons

We sincerely thank the reviewer for the careful assessment and for recognizing the motivation and experimental coverage of our work. We particularly appreciate the constructive suggestion to isolate the contribution of REINFORCE through behavior-cloning baselines and multi-seed evaluation.

Q1 & W1. How much improvement is specifically attributable to REINFORCE?

To isolate the contribution of policy-gradient optimization, we conducted two additional experiments.

First, we introduced BC-only, retaining the original behavior-cloning initialization while removing REINFORCE. This evaluates the effectiveness of supervised policy imitation alone.

Second, we introduced BC-only (Matched) to control for the additional training received by the full RL policy. We extended supervised BC training to match the overall policy-training budget of the full method, allowing us to distinguish the effect of the RL objective from simply performing more training.

We also extended the comparison with greedy selection to all three datasets.

Additional comparison (mIoU ↑):

DatasetGreedyBC-onlyBC-only (Matched)Full RL
Charades59.4159.4559.6559.82
ActivityNet59.6559.6359.8760.18
QVHighlights71.31±0.1271.08±0.4271.34±0.2971.52±0.07

QVHighlights reports mean±standard deviation over three independent end-to-end seeds, including verifier retraining; Charades and ActivityNet report single runs. The original Table 5 values (71.35 greedy; 71.55 full) correspond to one run.

Additional BC training improves performance but does not eliminate the RL advantage. Compared with the original BC-only policy, matched BC improves mIoU by 0.20, 0.24, and 0.26 points across the three datasets. Full RL nevertheless achieves further improvements of 0.17, 0.31, and 0.18 points, respectively.

Full RL also outperforms greedy selection by 0.41, 0.53, and 0.21 mIoU points. On QVHighlights, the full policy shows lower run-to-run variation (±0.07) than greedy (±0.12) and matched BC (±0.29).

These results provide evidence of an incremental contribution from REINFORCE beyond both verifier-guided greedy selection and supervised training.

The distinction is also reflected in the learning objectives. BC imitates reference-guided edit choices at individual states, whereas REINFORCE optimizes trajectory-level rewards incorporating realized utility changes, penalties for harmful and unnecessary edits, and final prediction quality. This directly supports learning sequential editing and stopping decisions.

We appreciate the reviewer's suggestion, which has enabled a more controlled assessment of the policy-gradient contribution.

AuthorsOpenReview ↗

High-IoU Revision Reliability and OMTG Split Ablation

We sincerely thank the reviewer for the thoughtful questions regarding high-quality prediction reliability and the contribution of split edits. These suggestions motivated two additional analyses that clarify how selective refinement affects accurate predictions and multi-event grounding.

Q2 & W2. Why are initially high-quality predictions sometimes degraded?

The degradation percentages in Table C.3 are conditioned on predictions that both have initial IoU ≥ 0.9 and undergo revision. They should not be interpreted as degradation rates among all initially accurate predictions.

Following the reviewer's suggestion, we computed the average magnitudes of positive and negative changes, together with the net IoU effect.

Additional high-IoU analysis (IoU in [0, 1]):

DatasetInitial IoU ≥ 0.9RL RevisedMean GainMean LossNet ΔIoU (Revised)
QVHighlights6901610.0750.030+0.010
Charades7481230.0680.046−0.016
ActivityNet1,2763550.0800.025+0.009

Mean Gain and Mean Loss are averaged over improved and degraded revised predictions, respectively. Net ΔIoU includes revised predictions whose IoU remains unchanged.

These results clarify why degradation frequency alone does not fully characterize reliability. Although Table C.3 reports more degraded than improved predictions among revised high-IoU samples, their average improvement magnitudes exceed their average degradation magnitudes. Consequently, the revised subsets achieve positive net IoU changes on QVHighlights and ActivityNet, while Charades retains a negative net change.

More importantly, we additionally evaluate the impact on all initially high-IoU predictions, assigning zero IoU change to predictions that were not revised. This captures the combined effects of revision frequency and revision quality.

Population-level comparison with greedy selection:

DatasetRL Revision RateGreedy Revision RateRL Net ΔIoU ↑Greedy Net ΔIoU ↑
QVHighlights23.33%38.12%+0.0023−0.0126
Charades16.44%13.90%−0.0026−0.0083
ActivityNet27.82%44.20%+0.0024−0.0319

Both methods are evaluated on the same initial high-IoU populations. Greedy revises 263, 104, and 564 predictions on QVHighlights, Charades, and ActivityNet, respectively.

Full RL achieves better population-level net IoU changes than greedy on all three datasets. On QVHighlights and ActivityNet, the full policy achieves positive net changes, whereas greedy produces negative changes. On Charades, the full policy revises slightly more high-IoU predictions but substantially reduces the overall net degradation.

These results show that the risk-aware mechanism mitigates damage through the combined effects of edit selection and revision quality. Its objective is to reduce harmful revisions rather than guarantee that every accepted edit is beneficial.

Q3. How much of the OMTG improvement comes from split edits?

We appreciate the suggestion to distinguish event separation from boundary refinement. We therefore evaluated SeTeR with split edits disabled, preserving the original QVHighlights-to-OMTG transfer setting without any OMTG training.

Additional OMTG ablation:

MetricTimeLens2SeTeR w/o SplitFull SeTeR
C-Acc ↑16.8816.8825.00
[email protected] ↑49.0448.9558.26
[email protected] ↑39.3039.2149.42
[email protected] ↑26.7426.8535.94
tIoU ↑48.9349.0051.56
EtF1 ↑13.2113.2419.06

Without split edits, the number of predicted intervals remains unchanged, so C-Acc stays at the baseline value of 16.88. The remaining boundary-only modifications yield results close to the original TimeLens2 predictions.

By contrast, restoring split edits increases C-Acc to 25.00, [email protected] to 49.42, and EtF1 to 19.06. Relative to the no-split variant, these correspond to improvements of 8.12, 10.21, and 5.82 points, respectively.

The results demonstrate that the OMTG improvements are primarily associated with the ability to revise event structure, rather than merely adjusting the boundaries of an unchanged number of intervals. In particular, split edits enable the model to separate occurrences that were initially merged into a single interval.

This directly supports our central motivation for refining the complete interval set, including both temporal boundaries and event structure.

We sincerely appreciate the reviewer's constructive suggestions, which have strengthened the empirical analysis of our framework.

AuthorsOpenReview ↗

Kind Reminder

Dear Reviewer,

We hope you are having a good week. Thank you again for your insightful comments. To address your questions, we have added controlled evaluations of REINFORCE, analyzed net IoU changes for initially accurate predictions, and conducted an OMTG ablation to isolate the contribution of split edits.

As the discussion deadline draws closer, we would greatly appreciate hearing whether these analyses address your concerns. Please let us know if further details or experiments would be useful; we would be happy to follow up while time remains for discussion.

If you find that the additional results have addressed your concerns, would you kindly consider raising your rating for our paper?

Thank you very much for your time and consideration. We look forward to your feedback.

AuthorsOpenReview ↗

Final Revision: RL Controls, Revision Reliability, and Split Edits

Thank you for your suggestions. We have updated the final version of the paper to include controlled RL comparisons, additional high-IoU reliability analyses, and a no-split ablation.

  • Contribution of REINFORCE. Over three independent end-to-end seeds on QVHighlights, including verifier retraining, full RL achieves 71.52±0.07 mIoU, compared with 71.31±0.12 for greedy, 71.08±0.42 for BC-only, and 71.34±0.29 for budget-matched BC.
  • Initially accurate predictions. We report conditional improvement and degradation magnitudes, together with net changes over all initially high-IoU predictions. On QVHighlights, Charades, and ActivityNet, RL produces population-level net IoU changes of +0.0023, −0.0026, and +0.0024, compared with −0.0126, −0.0083, and −0.0319 for greedy, respectively. These changes use the [0,1] IoU scale.
  • Contribution of split edits. On OMTG, disabling splits yields 16.88 C-Acc, 39.21 [email protected], and 13.24 EtF1. Full SeTeR achieves 25.00, 49.42, and 19.06, respectively, supporting the contribution of event separation.
Reviewer_V61JOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents SeTeR, a selective temporal refinement framework for video grounding. SeTeR formulates temporal refinement as a sequential decision-making process by employing a verifier to estimate the quality of candidate modifications and a reinforcement learning policy to guide refinement decisions with the underlying video grounding model remains frozen. Experiments demonstrate improvements across several temporal grounding benchmarks, along with additional results on a related temporal understanding task.

Critical strengths

(1) A meaningful reformulation of temporal refinement. SeTeR explicitly models the possibility that a proposed update may worsen the prediction and shifts attention from generating better candidate intervals to deciding whether a candidate modification should be applied. (2) Selective refinement based on estimated utility is a generalizable research direction. (3) The method preserves the pretrained grounding model. (4) The paper investigates alternatives to reinforcement learning. The comparison with threshold-based verification, direct gain regression, and greedy selection is valuable.

Critical weaknesses

(1) The necessity of reinforcement learning is not sufficiently demonstrated. This is my main concern. On QVHighlights, The reinforcement learning policy improves mIoU by only 0.20 points over greedy selection. (2) The overall performance gains are relatively modest. The reported improvements over the frozen grounding backbone are approximately 1–2 mIoU points across the main benchmarks. While these gains are consistent, their practical significance should be assessed against the additional inference-time computation. The paper reports increased inference cost, and a more detailed accuracy–latency analysis would help establish whether selective refinement offers a favorable trade-off.

Questions for authors

(1) Why does reinforcement learning outperform greedy selection by only a small margin? Is this difference statistically significant? (2) What proportion of proposed refinements is accepted, rejected, or terminated at different stages

4 comments
AuthorsOpenReview ↗

Title: RL Contribution Beyond Greedy and Behavior Cloning

We particularly appreciate the reviewer's questions regarding the incremental benefit of RL, its stability, and the practical accuracy–latency trade-off.

Q1 & W1. Contribution of RL beyond greedy and supervised selection.

We included greedy selection as a strong comparison to isolate the benefit of sequential policy learning. Greedy already exploits the learned verifier, while RL additionally optimizes edit selection and stopping decisions.

Following the reviewer's valuable suggestion, we conducted two additional experiments to progressively isolate the contribution of REINFORCE.

First, we introduced BC-only. This baseline retains the original behavior-cloning initialization but removes subsequent REINFORCE optimization. It therefore measures the performance achievable through supervised policy imitation alone.

Second, we introduced BC-only (Matched). Since the full RL policy receives additional training after behavior cloning, comparing it only with the original BC-only model could conflate the effect of the optimization objective with that of additional training. We therefore extended supervised BC training to match the overall policy-training budget of the full method. This provides a more controlled comparison between continued supervised imitation and reinforcement learning.

We further extended the evaluation to all three datasets.

Additional comparison (mIoU ↑):

DatasetTimeLens2GreedyBC-onlyBC-only (Matched)Full RL
Charades58.6059.4159.4559.6559.82
ActivityNet58.6059.6559.6359.8760.18
QVHighlights70.2071.31±0.1271.08±0.4271.34±0.2971.52±0.07

QVHighlights reports mean±standard deviation over three independent end-to-end seeds, including verifier retraining; Charades and ActivityNet report single runs. The original Table 5 values (71.35 greedy; 71.55 full) correspond to one run.

The additional supervised training is beneficial, but does not eliminate the advantage of RL. Extending BC training improves mIoU by 0.20, 0.24, and 0.26 points across Charades, ActivityNet, and QVHighlights, respectively. Nevertheless, full RL further outperforms this matched BC baseline by 0.17, 0.31, and 0.18 points.

Full RL also exceeds greedy selection by 0.41, 0.53, and 0.21 mIoU points, respectively. On QVHighlights, it additionally exhibits lower run-to-run variation (±0.07) than greedy (±0.12) and matched BC (±0.29).

These results distinguish the contribution of policy-gradient optimization from both verifier-guided greedy selection and additional supervised training.

Why does RL provide a further improvement? Behavior cloning learns to imitate reference-guided edit choices at individual states. In contrast, REINFORCE directly optimizes trajectory-level rewards that account for realized utility changes, penalties for harmful or unnecessary edits, and the final prediction quality. This enables the policy to optimize the consequences of sequential editing and stopping rather than supervised imitation alone.

The relatively small QVHighlights margin is understandable because greedy selection already exploits the learned verifier and captures much of the immediate edit utility. RL provides an additional improvement on top of this strong foundation.

Regarding inference cost, both strategies share the expensive candidate-generation and verification stages. Appendix C.3 reports that final selection accounts for only approximately 1–2 minutes of the complete evaluation.

We sincerely appreciate the reviewer's constructive suggestions, which have enabled a more controlled evaluation of the independent contribution of reinforcement learning.

AuthorsOpenReview ↗

Stage-Wise Decisions and Accuracy–Latency Trade-off

We thank the reviewer for the valuable questions regarding refinement behavior and computational efficiency. We provide the following clarification and additional analysis.

Q2. Acceptance, rejection, and termination across stages.

Our policy chooses an eligible edit or STOP at each stage. Candidates failing the revision threshold are excluded; there is no separate learned REJECT action. We therefore distinguish accepted edits, explicit STOP decisions, and termination when no eligible edit remains.

Stage-wise decision statistics:

DatasetStageReachedEdit (%)STOP (%)No Eligible (%)
QVHighlights11,54140.645.314.1
262525.651.023.4
316018.856.325.0
Charades13,36347.033.819.2
21,58023.155.921.0
336516.758.125.2
ActivityNet14,50039.842.417.8
21,79023.554.522.0
342016.956.726.4

Percentages are conditioned on reaching each stage. Across all three datasets, edit acceptance decreases while explicit stopping becomes more frequent at later stages. This behavior is consistent with selective refinement: the policy increasingly preserves the current prediction as revision proceeds. Reaching three accepted edits also terminates refinement.

W2. Accuracy–latency trade-off.

As reported in Appendix C.3, TimeLens2 requires 6 h 20 min, while SeTeR requires 7 h 23 min, corresponding to 16.6% additional inference time on two A100 GPUs.

Of the additional 63 minutes, candidate generation accounts for 28 minutes, verification for 33 minutes, and final selection for only 1–2 minutes.

For this overhead, SeTeR improves mIoU by 1.22, 1.58, and 1.35 points, and [email protected] by 1.95, 1.65, and 2.27 points on Charades, ActivityNet, and QVHighlights, respectively.

Thus, most additional computation is devoted to obtaining and verifying local visual evidence. The learned policy operates on these verifier outputs, making selection a relatively small part of the overall inference cost.

We sincerely appreciate the reviewer's constructive suggestions, which help clarify the practical behavior and computational trade-off of our approach.

AuthorsOpenReview ↗

Kind Reminder

Dear Reviewer,

We hope you are doing well. We sincerely appreciate your thoughtful feedback. In our responses, we have further examined the contribution of RL through controlled comparisons and provided stage-wise decision statistics and a detailed accuracy–latency analysis.

With the discussion period nearing its end, we would be grateful if you could share any remaining questions or concerns. We welcome the opportunity to provide further clarifications or additional experiments, and an early response would give us more time to complete these follow-ups.

If our responses and additional evidence have resolved your concerns, we would appreciate your consideration of a higher rating.

Thank you again for your time and thoughtful consideration. We look forward to hearing from you.

AuthorsOpenReview ↗

Final Revision: RL Evaluation and Compute Costs

Thank you for your suggestions. We have updated the final version of the paper to include controlled policy comparisons, stage-wise decision statistics, and a detailed runtime breakdown.

  • Contribution of RL. Across three independent end-to-end seeds on QVHighlights, including verifier retraining, full RL achieves 71.52±0.07 mIoU, exceeding greedy selection by 0.21 points and budget-matched BC by 0.18 points. The additional single-run comparisons show gains over greedy of 0.41 points on Charades and 0.53 points on ActivityNet.
  • Stage-wise decisions. Conditional edit acceptance decreases from the first to the third stage: 40.6%→18.8% on QVHighlights, 47.0%→16.7% on Charades, and 39.8%→16.9% on ActivityNet. Explicit stopping becomes more frequent at later stages.
  • Compute costs. Evaluation takes 7 h 23 min, compared with 6 h 20 min for TimeLens2, corresponding to 16.6% overhead. Additional candidate generation takes 28 minutes, verification takes 33 minutes, and final selection takes approximately 1–2 minutes.
Reviewer_knhkOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a Selective Temporal Reinforcement Learning framework for temporal grounding. The method generates candidate temporal revisions through local video re-observation and introduces a learnable verifier to estimate the potential improvement, degradation risk, and positive gain of each revision. A reinforcement learning policy is then trained to select beneficial edits or stop refinement to avoid unnecessary modifications. Experiments on four datasets demonstrates the effectiveness of the method.

Critical strengths
  1. The overall framework is logically designed and follows a clear refinement pipeline. It keeps the base grounder frozen, generates candidate revisions through local video re-observation, and employs a learned policy to decide whether to accept a revision or stop refinement.
  2. Experiments on three temporal grounding benchmarks and a transfer evaluation on OMTG demonstrate consistent improvements in temporal localization accuracy and multi-event grounding performance.
Critical weaknesses
  1. The limited performance improvement. As shown in Table 5, the full policy improves mIoU by only 0.20 points over the greedy counterpart on QVHighlights, without variance estimates or a behavior-cloning-only baseline. It remains unclear whether policy-gradient optimization offers meaningful advantages over simpler supervised edit selection. The authors are encouraged to include a behavior-cloning-only baseline, report results across multiple random seeds, and extend the RL versus greedy comparison to the other two datasets to better demonstrate the effectiveness of the RL policy.
  2. Insufficient justification for the three-component revision verifier. All three prediction targets are derived from the same utility difference (\Delta), raising concerns about potential redundancy. Although Table 5 includes a comparison with direct gain regression, component-wise ablations are missing. The authors should evaluate the individual contributions of the improvement, degradation, and gain prediction heads to justify the multi-head design.
Questions for authors

Please refer to the weakness part.

4 comments
AuthorsOpenReview ↗

Cross-Dataset RL Gains and Matched Behavior-Cloning Controls

We sincerely thank the reviewer for the careful assessment and constructive suggestions. We particularly appreciate the recommendation to include behavior-cloning baselines, multi-seed evaluation, and RL–greedy comparisons across all three benchmarks.

Q1 & W1. Does reinforcement learning provide meaningful benefits beyond simpler edit selection?

We included greedy selection as a strong baseline to examine the incremental value of learned sequential decisions. Both greedy and RL use the same three-component verifier, while RL additionally learns how to select edits and when to terminate refinement.

Following the reviewer's suggestion, we conducted two additional experiments to isolate the contribution of REINFORCE.

(1) BC-only: We retain the original two-epoch behavior-cloning initialization but remove subsequent REINFORCE optimization. This evaluates the effectiveness of supervised policy imitation alone.

(2) BC-only (Matched): Since the full policy receives additional training after behavior cloning, we further extend supervised BC training to match the overall policy-training budget of the RL method. This controls for the possibility that performance differences arise merely from additional training rather than the RL objective.

We also extended the RL–greedy comparison to Charades and ActivityNet.

Additional results (mIoU ↑):

DatasetTimeLens2GreedyBC-onlyBC-only (Matched)Full RL
Charades58.6059.4159.4559.6559.82
ActivityNet58.6059.6559.6359.8760.18
QVHighlights70.2071.31±0.1271.08±0.4271.34±0.2971.52±0.07

QVHighlights reports mean±standard deviation over three independent end-to-end seeds, including verifier retraining. Charades and ActivityNet report single runs. The original Table 5 values (71.35 greedy; 71.55 full) correspond to one run.

First, RL consistently outperforms greedy in the reported comparisons. Beyond the original 0.20-point improvement on QVHighlights, the extended evaluation shows gains of 0.41 mIoU on Charades and 0.53 on ActivityNet. The three-seed QVHighlights results yield a mean advantage of 0.21 points.

Second, the RL advantage is not explained solely by additional training. Extending BC training improves performance by 0.20, 0.24, and 0.26 mIoU points across the three datasets. Nevertheless, full RL further improves upon matched BC by 0.17, 0.31, and 0.18 points, respectively.

Third, RL exhibits lower variation on QVHighlights. Its standard deviation is 0.07, compared with 0.12 for greedy and 0.29 for matched BC. These results quantify run-to-run variation without relying on an unsupported claim of formal statistical significance.

The incremental advantage is also consistent with the difference in training objectives. BC imitates reference-guided edit choices at individual states, whereas REINFORCE directly optimizes trajectory-level rewards that incorporate realized utility changes, penalties for harmful and unnecessary edits, and final prediction quality. This allows the policy to optimize sequential editing and stopping decisions.

Regarding inference cost, both strategies share the expensive candidate-generation and verification stages. As quantified in Appendix C.3, final selection itself accounts for approximately 1–2 minutes of the complete evaluation.

Together, the additional experiments provide stronger evidence for the contribution of RL beyond greedy selection and supervised imitation. We sincerely appreciate the reviewer's valuable suggestions.

AuthorsOpenReview ↗

Three-Head Verifier: Design Rationale and Component Ablations

We sincerely thank the reviewer for the insightful question regarding the necessity of our three-component verifier. This is an important point for clarifying why we separately model improvement, degradation, and positive gain.

Q2 & W2. Why are three verifier components necessary if they share the same utility difference?

All three supervision targets originate from the same utility difference Δ. This is intentional: they capture complementary decision-relevant properties rather than independent supervision sources.

Our design rationale can be understood through three steps.

(1) A single gain estimate does not fully characterize uncertainty.

A natural starting point is to regress the utility difference Δ and rank candidate revisions accordingly. The direct gain regression baseline in Table 5 achieves 71.25 mIoU, compared with 71.55 for our full method.

However, a single point estimate does not separately quantify the probability of a harmful outcome. Two revisions with similar predicted utility changes may have different downside risks.

(2) Explicit degradation prediction makes downside risk visible.

We therefore introduce a degradation head estimating the probability that an edit decreases utility beyond the margin δ=0.01. This enables the decision mechanism to explicitly penalize harmful-revision risk rather than relying solely on a scalar gain estimate.

(3) Improvement probability distinguishes meaningful improvements from near-neutral changes.

A low degradation probability does not necessarily imply a high probability of improvement: an edit may simply leave the prediction nearly unchanged.

Our improvement head separately estimates the probability that Δ exceeds +0.01. Since changes within [−0.01, 0.01] are treated as neutral, improvement and degradation probabilities are not simple complements.

Meanwhile, the positive-gain head estimates the magnitude of beneficial changes, distinguishing edits with similar improvement probabilities but different potential rewards.

Accordingly, our decision score combines all three predictions:

A(e)=pimp(e)g+(e)−λriskpdeg(e).A(e)=p_{\mathrm{imp}}(e)g_+(e)-\lambda_{\mathrm{risk}}p_{\mathrm{deg}}(e).

The improvement and gain terms jointly prioritize promising revisions, while the degradation term explicitly penalizes harmful-revision risk. We do not claim that the three targets are statistically independent; their purpose is to capture complementary properties of the utility change.

Following the reviewer's suggestion, we conducted component-wise ablations to quantify their individual contributions.

Additional verifier ablations on QVHighlights:

VerifiermIoU ↑Revision RateHarmful Rate ↓
Full three-head verifier71.5523.7%33.4%
w/o Improvement71.3925.7%39.8%
w/o Degradation71.0830.2%41.0%
w/o Gain71.3025.5%38.2%

Harmful Rate denotes the fraction of actually revised predictions whose IoU decreases.

Removing any component reduces mIoU and increases the harmful-revision fraction. Removing degradation prediction produces the largest mIoU reduction (−0.47) and raises the harmful rate from 33.4% to 41.0%.

Removing improvement probability or positive gain also reduces localization accuracy and increases harmful revisions. These results provide direct empirical evidence that the three components play complementary roles in the proposed decision mechanism.

Together, the design rationale and component-wise evidence justify separating improvement likelihood, degradation risk, and positive gain despite their shared underlying supervision.

We sincerely appreciate the reviewer's constructive suggestion and welcome further questions or discussion.

AuthorsOpenReview ↗

Kind Reminder

Dear Reviewer,

We hope this message finds you well. Thank you again for your constructive suggestions. Following your feedback, we have added behavior-cloning controls, multi-seed and cross-dataset comparisons, and component-wise ablations of the three-head verifier.

As the discussion deadline approaches, we would greatly appreciate your feedback on these additional results. Hearing from you soon would give us more time to clarify any remaining questions and conduct further experiments you consider helpful.

If these additions have addressed your concerns, would you kindly consider raising your rating for our paper?

Thank you for your time and consideration. We look forward to continuing the discussion.

AuthorsOpenReview ↗

Final Revision: RL Controls and Verifier Ablations

Thank you for your suggestions. We have updated the final version of the paper to include additional policy comparisons and component-wise verifier ablations.

  • RL controls and multiple seeds. Over three independent end-to-end seeds on QVHighlights, including verifier retraining, full RL achieves 71.52±0.07 mIoU, compared with 71.31±0.12 for greedy selection, 71.08±0.42 for BC-only, and 71.34±0.29 for budget-matched BC. On Charades and ActivityNet, the additional single-run comparisons show gains of 0.41 and 0.53 points over greedy, respectively.
  • Verifier ablations. Removing the improvement, degradation, or gain head reduces mIoU from 71.55 to 71.39, 71.08, and 71.30, respectively. All three ablations also increase the harmful-revision fraction, with the largest accuracy reduction occurring when degradation prediction is removed.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Final Summary: Reviewer Feedback and Additional Evidence

Dear Reviewers, ACs, SACs, and PCs,

We sincerely thank all three reviewers for their careful evaluations and constructive suggestions. Following their feedback, we conducted additional controlled comparisons and analyses and incorporated these results into the revised manuscript.

Our main contribution is SeTeR, a framework that learns whether temporal revisions should be applied to the predictions of a frozen video grounder. A plausible candidate is not necessarily better than the current answer. SeTeR therefore evaluates the benefit and degradation risk of edits relative to the complete interval set, then learns to select edits or stop. This formulation accounts for both temporal boundaries and event structure while preserving the pretrained grounder.

Strengths noted by the reviewers

  • A meaningful formulation of selective refinement. V61J highlighted the explicit treatment of potentially harmful updates and the shift from generating candidate intervals to deciding whether a modification should be applied. WtiC similarly recognized the value of evaluating changes in grounding quality rather than candidate quality alone.
  • A clear framework with consistent empirical gains. knhk appreciated the refinement pipeline and the frozen-grounder design, together with consistent improvements across three temporal grounding benchmarks and transfer to OMTG.
  • Comparisons and analyses that explain revision behavior. V61J valued the comparisons with threshold-based verification, direct gain regression, and greedy selection. WtiC appreciated the experimental coverage, including ablations, oracle results, runtime measurements, and analyses of successful and failed revisions.

Additional evidence provided during the discussion

  • RL contribution beyond supervised controls. Across three independent end-to-end seeds on QVHighlights, full RL achieves 71.52±0.07 mIoU, compared with 71.31±0.12 for greedy, 71.08±0.42 for BC-only, and 71.34±0.29 for budget-matched BC. Additional single-run comparisons show gains over greedy of 0.41 points on Charades and 0.53 points on ActivityNet.
  • Complementary verifier components. Removing the improvement, degradation, or gain head reduces mIoU from 71.55 to 71.39, 71.08, and 71.30, respectively, while increasing harmful-revision fractions.
  • Revision reliability and event separation. We report gain/loss magnitudes and population-level net changes for initially accurate predictions. On OMTG, enabling split edits improves C-Acc from 16.88 to 25.00 and EtF1 from 13.24 to 19.06 relative to the no-split variant.
  • Decision behavior and computational cost. We provide stage-wise editing and stopping statistics. Total inference overhead is 16.6%, with final selection accounting for approximately 1–2 minutes.

We appreciate the reviewers’ suggestions, which helped strengthen the empirical assessment of SeTeR. We respectfully ask that the additional evidence and revised manuscript be considered in the final evaluation.

Best regards, The authors of Paper 32012

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.