acceptodds
Under review as a conference paper at ICLR 2027

Training-Free Video Paragraph Grounding with Video Sentence Localizers via Sequence Consistency

Abstract

Video Paragraph Grounding (VPG) jointly localizes the temporal segments described by all sentences in a paragraph. Although paragraph-level descriptions provide semantic context and temporal relations for disambiguating repeated or visually similar events in long videos, existing VPG models can still underperform strong video sentence localizers. Through a diagnostic analysis of existing VPG methods, we find that the sequential organization of paragraph sentences provides a simple yet effective global cue for coordinating independently grounded segments. Based on this insight, we propose Sentence-to-Paragraph Grounding (S2PG), a lightweight and training-free framework that reformulates VPG as a sequence-level candidate selection problem. S2PG first obtains multiple candidate segments and confidence scores for each sentence from an off-the-shelf Video Sentence Grounding (VSG) model, and then applies Sequence-Aware Global Optimization (SAGO) to jointly optimize sentence-level grounding confidence and paragraph-level temporal consistency. Experiments on two representative VPG benchmarks demonstrate that S2PG consistently improves strong VSG baselines and achieves state-of-the-art performance among existing VPG methods.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Video Paragraph Grounding, Training-Free

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_japhOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper illustrates a new method, Sentence-to-Paragraph Grounding (S2PG) which is a lightweight model reformulating Video Paragraph Grounding (VPG) as a sequence-level candidate selection problem and improves the performance.

Critical strengths

The method is easy and effective on the dataset present in the paper.

Critical weaknesses

The test sets are limited and more experiments should be done on more datasets for solid experiments and conclusion.

Questions for authors

How does this paper inspire related works and further study? Could you please provide more visualization results on how this method outperforms others?

3 comments

Weakness

We provide additional results on ActivityNet Captions and Charades-CD in Tables 1–3 of this response. On ActivityNet Captions, S2PG improves SnAG across all reported metrics, increasing mIoU from 46.71% to 49.03% (+2.32 percentage points). This further supports the effectiveness of sequence-consistent candidate selection.

Table 1. ActivityNet Captions grounding results (%).

Method[email protected][email protected][email protected]mIoU
SnAG63.6247.9830.2346.71
S2PG(SnAG)67.2550.6631.5149.03

Following SVPTR, we additionally evaluate S2PG on Charades-CD. S2PG improves SnAG across all reported metrics on both the IID and OOD splits, increasing mIoU from 46.15% to 48.00% and from 36.50% to 38.66%, respectively. These gains support the effectiveness of sequence-consistent candidate selection under this setting.

Table 2. Charades-CD IID(%).

Method[email protected][email protected][email protected]mIoU
PRVG†66.1046.7822.1143.59
SnAG63.4354.3133.2946.15
S2PG(SnAG)66.1056.5034.5148.00

Table 3. Charades-CD OOD grounding results(%).

Method[email protected][email protected][email protected]mIoU
PRVG†54.3631.3814.4135.26
SVPTR55.1432.4415.5336.01
SiamGTR59.0735.4714.9538.87
SnAG53.3838.8220.7036.50
S2PG(SnAG)56.7641.7021.5638.66

PRVG† results on Charades-CD is reproduced with the released code. S2PG also achieves the highest [email protected] and [email protected] scores among the listed methods on both Charades-CD splits.

Question

  • Research Contributions and Significance

Our perturbation experiments can inform future VPG research by examining how existing methods exploit different types of paragraph information. These findings can guide future studies in identifying underutilized temporal-order cues and exploring how to use them to improve localization performance.

We then identify paragraph-level temporal order as an effective yet underexplored cue for connecting video sentence grounding (VSG) with paragraph-level joint grounding. Building on this finding, we propose S2PG, a lightweight, training-free framework that can be directly combined with existing VSG models. S2PG explores a new way to exploit temporal-order information by reformulating VPG as a sequence-level candidate-selection problem that balances sentence-level localization confidence with paragraph-level sequence consistency. This perspective can inspire future research toward more powerful VPG models.

Finally, S2PG (OSGNet) surpasses the previous VPG SOTA by 14.84% in mIoU on TACoS and 11.21% in [email protected] on Ego4D-GoalStep. S2PG can therefore serve as a strong VPG baseline for practical use and comparisons with future methods.

  • Additional Qualitative Comparisons

To address your request for additional visualizations showing how our method outperforms other methods, the figure available at this link presents a qualitative comparison of SnAG, S2PG (SnAG), and PRVG on a Charades-CD paragraph. SnAG predicts an earlier start for Q3 than for Q2, which is inconsistent with the temporal order of the paragraph. By penalizing start-time inversions during candidate reranking, S2PG selects a later candidate for Q3, restoring temporal consistency and yielding a closer match to the ground truth. Compared with PRVG, S2PG (SnAG) also produces predictions that align more closely with the ground truth, particularly for Q3 and Q4. Qualitative comparisons on TACoS and Ego4D-GoalStep are already included in Figure 5 and Figure 7 of the manuscript, respectively.

Thank you for your clear answers. The statistical and visualization results provide more details about the method.

Thank you for your positive reassessment. We sincerely appreciate your helpful feedback, which has greatly helped us improve the paper.

Reviewer_muWdOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes S2PG, a training-free framework for video paragraph grounding using existing video sentence localizers. It generates candidate segments independently for each sentence and applies dynamic programming to jointly maximize grounding confidence while penalizing start-time inversions. Experiments on TACoS and Ego4D-GoalStep demonstrate substantial improvements over the underlying localizers and competitive VPG methods.

Critical strengths
  1. The framework is simple and practical, introducing paragraph-level coordination without retraining or modifying the underlying localizer.

  2. The improvements are substantial: S2PG increases mIoU by 7.45 points over SnAG on TACoS and 8.21 points over OSGNet on Ego4D-GoalStep.

  3. Exact optimization, negligible reported reranking overhead, and ablations on candidate count, confidence costs, and selection strategies support the method’s practical usefulness.

Critical weaknesses
  1. SAGO uses standard dynamic programming for a chain-structured objective comprising confidence scores and an adjacent-order penalty. Comparisons with greedy and beam search demonstrate the benefit of exact optimization, but do not establish the proposed objective’s advantage over simpler alternatives, particularly hard-monotonic decoding using identical candidate proposals.

  2. The context perturbation analysis examines only DepNet and PRVG, while stronger VPG methods are absent. Moreover, the additional benefit of target-relative ordering is negligible on Ego4D-GoalStep (Table 5). These findings support observations about the evaluated models, but provide limited evidence for broader conclusions about how existing VPG methods exploit paragraph context.

  3. The flexibility of the soft order prior is insufficiently validated. Appendix C.1 reports nondecreasing ground-truth starts for 100% and 98.32% of adjacent query pairs on the two benchmarks. With timestamps measured in seconds, confidence scores in [0,1], and λ = 20, the objective strongly penalizes inversions. Evaluation on examples with genuine order violations would help establish whether valid deviations are preserved in practice.

  4. S2PG can only select from the underlying localizer’s proposals. Although candidate-set size is studied, the paper does not report top-K oracle recall or oracle mIoU. These measurements would clarify how much available improvement SAGO captures and whether remaining failures arise from missing proposals or incorrect sequence selection.

  5. The reranker uses candidate confidence and start times, without directly modeling cross-sentence semantic dependencies or end-time relationships. Although the limitation section acknowledges unused semantic context, the experiments do not isolate cases requiring such information. Failure analysis on repeated events, cross-sentence references, and ambiguous overlapping segments would clarify when sequence consistency alone is insufficient.

Questions for authors

The incremental algorithmic contribution, missing comparisons with simple order-constrained decoding, and limited diagnostic coverage leave its contribution insufficiently established.

7 comments

Weakness 1

Thank you for your appreciation of our dynamic programming algorithm. As noted, our objective consists of a candidate-confidence term and an adjacent-order penalty. The suggested hard-monotonic decoder is not explicitly defined. If it considers only temporal order and treats all candidates equally, multiple feasible paths can remain, and an additional criterion is required to distinguish their quality. If higher-ranked or higher-confidence candidates are preferred, the decoder still combines candidate quality with temporal-order consistency which is consistent with SAGO. A claim that this alternative is simpler therefore requires an explicit objective and candidate-selection rule.

Weakness 2

Regarding the choice of models for the perturbation analysis, we selected DepNet and PRVG as representative VPG baselines because their code is publicly available, allowing us to conduct the analysis. We did not include the stronger HSCNet because its code has not been publicly released.

Table 1. Target-relative ordering analysis for PRVG on ActivityNet Captions (%).

Input VariantmIoU[email protected][email protected][email protected]
Target-Fixed Context Shuffle53.3675.9457.7934.50
Within-Side Context Shuffle55.4978.2060.8837.45

Table 2. Target-relative ordering analysis for DepNet on ActivityNet Captions (%).

Input VariantmIoU[email protected][email protected][email protected]
Target-Fixed Context Shuffle52.8473.0557.1235.31
Within-Side Context Shuffle53.2473.5357.6535.91

We agree with the reviewer that target-relative ordering provides negligible additional gains on Ego4D-GoalStep. One possible explanation is the difficulty of adapting these architectures to long-video grounding. Their performance degrades substantially on full-length videos and remains low under the clip setting used for the perturbation analysis, which may make the small differences between perturbations difficult to interpret. We therefore further examine target-relative ordering through perturbation experiments on ActivityNet Captions. Compared with Target-Fixed Context Shuffle, Within-Side Context Shuffle additionally preserves whether each surrounding sentence appears before or after the target. As shown in Tables 1 and 2 of this response, preserving these relations increases mIoU by 2.13 percentage points for PRVG and 0.40 points for DepNet. These results show that target-relative ordering can provide useful localization cues for the evaluated models, with its contribution varying across models and datasets.

Weakness 3

Table 3. ActivityNet Captions grounding results (%).

Method[email protected][email protected][email protected]mIoU
SnAG63.6247.9830.2346.71
S2PG(SnAG)67.2550.6631.5149.03

Our temporal-order assumption is supported by the annotation statistics even in the more natural storytelling setting of ActivityNet Captions. Among 12,073 adjacent query pairs within PRVG-defined paragraphs, 98.58% have nondecreasing ground-truth start times, while 172 pairs (1.42%) exhibit start-time inversions. These statistics support the temporal-order prior and confirm the presence of a small proportion of annotated order violations. Otherwise, as shown in the Table 3 of this response, S2PG improves mIoU by 2.32 points on ActivityNet Captions with SnAG, demonstrating its effectiveness.

Weakness 4

Thank you for pointing out the need to report top-K oracle results. Table 4 of this response compares the base VSG localizers, S2PG, and their top-20 oracles. For each query, the oracle selects the candidate with the highest IoU to the ground truth.

Table 4. Base VSG, S2PG, and top-20 oracle results on TACoS and Ego4D-GoalStep (%).

DatasetResult[email protected][email protected]mIoU
TACoSSnAG (R@1)55.5645.1441.69
TACoSS2PG (SnAG, R@1)65.5152.7949.14
TACoSSnAG (R@20)93.5386.5073.75
Ego4D-GoalStepOSGNet (R@1)44.2737.3734.49
Ego4D-GoalStepS2PG (OSGNet, R@1)54.9045.7942.70
Ego4D-GoalStepOSGNet (R@20)82.5077.0067.76

S2PG improves mIoU over the base localizers by 7.45% on TACoS and 8.21% on Ego4D-GoalStep. The remaining gaps to the oracle are 24.61% and 25.06%, respectively, indicating further room for improving candidate selection. Oracle recall below 100% at the reported IoU thresholds also shows that some queries lack a sufficiently accurate proposal in the top-20 set.

Weakness 5

Thank you for suggesting a more targeted analysis of challenging cases. S2PG primarily improves candidate selection through paragraph-level sequence consistency, together with the base localizer’s confidence scores, while S2PG does not explicitly use cross-sentence semantic context or end-time relationships to select among candidates or refine their boundaries.

Thank you again for your helpful comments. We have incorporated the following revisions into the manuscript.

  • W2. Clarified the model selection in Section 3.1 and added the full ActivityNet Captions perturbation analysis in Appendix A.1.
  • W3. Added ActivityNet Captions grounding results in Section 5.2 and Table 4, together with temporal-order statistics in Appendix C.1.
  • W4. Added top-20 oracle comparisons in Appendix C.6.
  • W5. Expanded the limitations discussion concerning candidate-boundary precision, contextual semantics, and end-time relationships in Appendix D.

The revised passages are highlighted in blue.

Reviewer_wnn9OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper addresses Video Paragraph Grounding (VPG), where existing dedicated models often perform worse than single-sentence localizers.

The authors analyze existing VPG models and find that temporal sentence ordering, not fine-grained inter-sentence semantics, serves as the primary coordination signal.

Based on this, they propose S2PG, a training-free framework that takes top-K segment candidates from an VSG model.

S2PG then uses DP to pick the best segment sequence by balancing individual confidence scores against pairwise start-time inversion penalties.

Across the TACoS and Ego4D-GoalStep benchmarks, S2PG boosts underlying VSG models and outperforms existing VPG baselines with minor computational overhead.

Critical strengths
  1. Instead of training large, end-to-end multimodal models to jointly encode long videos and entire paragraphs, S2PG frames paragraph grounding as a post-processing candidate-selection problem solved via exact dynamic programming in O(NK^2) time. It adds very little runtime overhead (0.21s on CPU for 4,001 queries on TACoS) while being model-agnostic.
  2. S2PG achieves decent gains across strong base localizers (SnAG and OSGNet) on both TACoS (+7.45% and +4.56% mIoU) and Ego4D-GoalStep (+8.21% mIoU over OSGNet), outperforming prior dedicated VPG methods.
Critical weaknesses
  1. The core sequence penalty penalizes start-time inversions, which inherently assumes that sentences in a paragraph strictly follow chronological order. While the authors show this holds for nearly 100% of the cases in procedural/cooking datasets (TACoS and Ego4D-GoalStep), this assumption breaks down in open-domain videos or natural storytelling that involve flashbacks, non-linear descriptions, or concurrent events.
  2. The other standard VPG benchmark used by DepNet, PRVG, and HSCNet evaluates on ActivityNet Captions. Because ActivityNet Captions features open-domain videos where start times are not strictly non-decreasing, it is unclear whether S2PG still outperforms dedicated VPG models like HSCNet and PRVG on general non-procedural VPG benchmarks.
  3. Because the framework operates entirely as a candidate selection mechanism over top-K proposals, it cannot correct or refine segment boundaries that the underlying VSG model poorly estimates. If the optimization does not retrieve the true moment within the top-K candidates, it cannot recover it.
Questions for authors
  • How sensitive is the method to paragraphs containing non-chronological narratives (e.g., retrospective statements like "He washed his hands after chopping the onions") or parallel events? Did you test performance on any non-procedural benchmark (such as ActivityNet Captions) where chronological ordering is less strict?
  • In Tables 2 and 3, how does the top-K oracle recall of the base VSG localizers compare to S2PG's performance? This will help clarify how much headroom remains in the candidate sets.
  • You state that S2PG is model-agnostic; however, you evaluate it only on dense, multi-scale anchor-free localizers (SnAG and OSGNet). How can S2PG be applied to generative MLLM-based localizers such as UniTime (Table 2) that directly decode a single timestamp pair rather than a calibrated top-K proposal list?
9 comments

Weakness 1 & Question 1

Actually, in the more natural storytelling setting of ActivityNet Captions, the temporal-order assumption still holds for the vast majority of adjacent annotated query pairs. Specifically, among 12,073 adjacent query pairs within PRVG-defined paragraphs in the evaluated test set, only 172 (1.42%) have inverted ground-truth start times; the remaining 98.58% have nondecreasing starts, including ties. This finding provides further empirical support for using start-time order as a soft temporal prior in S2PG.

Weakness 2

Table 1. ActivityNet Captions grounding results (%).

Method[email protected][email protected][email protected]mIoU
SnAG63.6247.9830.2346.71
S2PG(SnAG)67.2550.6631.5149.03
PRVG78.2761.1537.8355.62

On ActivityNet Captions, S2PG improves SnAG's [email protected] from 47.98% to 50.66% and mIoU from 46.71% to 49.03%, absolute gains of 2.68 and 2.32 percentage points, respectively. This shows that paragraph-level sequence information can complement independent sentence-level localization predictions: sequence-consistent optimization improves candidate selection and localization accuracy. Although our method still trails dedicated VPG models, the gap may be related to the relatively large fraction of video duration covered by ground-truth segments in ActivityNet; we discuss this in our response to Weakness 3.

Weakness 3

Table 2. Top-1 overlap categories and top-20 candidate overlap with ground truth (%).

DatasetBase VSGTop-1: no overlapTop-1: overlap, IoU < 0.5Top-20: none overlap with ground truth
ActivityNetSnAG11.35%40.67%0.03%
TACoSSnAG30.92%23.94%0.67%
Ego4D-GoalStepOSGNet46.11%18.44%9.90%

We agree that S2PG cannot refine proposal boundaries, however, our goal is to improve candidate selection using paragraph-level sentence-order consistency. As shown in Table 2 of this response, on TACoS and Ego4D-GoalStep, complete misses by the top-1 prediction are more common than imprecise localization of a partially overlapping prediction (0 < IoU < 0.5). Since the top-20 candidates include at least one segment overlapping the ground truth for most queries, many of these complete misses leave room for better candidate selection, further motivating our sequence-consistency optimization.

ActivityNet shows the opposite pattern, mainly because its ground-truth segments are much longer, making most top-1 predictions overlap with them. Previous work[1] has noted this problem. On average, ground-truth segments occupy 34.60% of video duration on ActivityNet, compared with 8.74% on TACoS and 2.23% on Ego4D-GoalStep.

[1] Yuan Y, Lan X, Wang X, et al. A closer look at temporal sentence grounding in videos: Dataset and metric[C]//Proceedings of the 2nd international workshop on human-centric multimedia analysis. 2021: 13-21.

Question 2

As shown in the Table 3 of this response, we compare each base VSG localizer, S2PG, and its top-20 oracle . At all metrics, the top-20 oracle exceeds S2PG on both datasets, indicating remaining headroom in the candidate sets.

Table 3. Base VSG, S2PG, and top-20 oracle results on TACoS and Ego4D-GoalStep (%).

DatasetResult[email protected][email protected]mIoU
TACoSSnAG (R@1)55.5645.1441.69
TACoSS2PG (SnAG, R@1)65.5152.7949.14
TACoSSnAG (R@20)93.5386.5073.75
Ego4D-GoalStepOSGNet (R@1)44.2737.3734.49
Ego4D-GoalStepS2PG (OSGNet, R@1)54.9045.7942.70
Ego4D-GoalStepOSGNet (R@20)82.5077.0067.76

Question 3

Thank you for pointing out that our wording was too broad. By model-agnostic, we mean that S2PG does not depend on the localizer's internal architecture; to apply it nontrivially, the localizer must output multiple scored temporal candidates per query.

We focus on enhancing specialized VSG localizers because they already provide competitive temporal grounding accuracy. On TACoS, OSGNet achieves 50.89 mIoU, compared with 50.00 for the MLLM-based UniTime. Otherwise, MLLM-based grounding of long videos can incur substantial computational and memory costs, constraining inference efficiency. Moreover, many existing generative localizers do not natively provide multiple candidate segments with corresponding confidence scores. Therefore, UniTime is included only as a comparison baseline, and we have not adapted S2PG to its single-pair output.

Thank you again for your constructive feedback. We have incorporated the following revisions into the manuscript.

  • W1 and Q1. Added temporal-order statistics on ActivityNet Captions in Appendix C.1.
  • W2. Added ActivityNet Captions grounding results in Section 5.2 and Table 4.
  • W3. Added candidate coverage and localization error analyses in Appendix C.5, and expanded the discussion of fixed-candidate coverage and boundary precision in Appendix D.
  • Q2. Added top-20 oracle comparisons in Appendix C.6. The revised passages are highlighted in blue.

Comments on the paper

2 comments

We have provided detailed responses to all reviewers’ questions and concerns. If our responses have addressed your concerns, we would appreciate it if you could consider raising your scores. If you have any remaining concerns, we would be very happy to discuss them further.

We have also evaluated S2PG with additional base localizers, with results included in Tables 2 and 3 of the revised manuscript. On TACoS, S2PG improves UniversalVTG’s mIoU from 48.74% to 53.07%. On Ego4D-GoalStep, it improves DeCafNet’s mIoU from 16.24% to 29.07% and UniversalVTG’s from 21.29% to 37.31%. Together with the existing results on SnAG and OSGNet, these additional experiments provide broader evidence that S2PG’s benefits extend across different base localizers, rather than being specific to a particular model.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.