acceptodds
Withdrawn from ICLR 2027

Dynamics-Guided Distribution Shaping for Reliable Vision-Language Grounding

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Vision-language grounding localizes visual regions described by natural language, yet current systems rarely indicate when their localization predictions are unreliable. Existing single-pass probabilistic regression methods mainly estimate uncertainty from the current prediction state and therefore overlook how sample-wise grounding responses evolve during optimization. We propose Dynamics-guided Distribution Shaping (DDS), which uses inter-epoch prediction variation between two consecutive epochs as a local training-dynamics signal. DDS converts this variation into a sample-specific reliability coefficient that guides bounded prediction-distribution shaping and strengthens residual correction for unstable samples. The dynamics branch is used only during training, preserving single-forward-pass localization and uncertainty estimation at inference. Across RefCOCOg and RefCOCO+ with three multimodal backbones, DDS consistently improves uncertainty–error alignment while maintaining competitive grounding performance. Under zero-adaptation transfer from RefCOCO+ to Ref-Adv-s, DDS further improves uncertainty reliability and out-of-distribution discrimination under semantic and reasoning shift.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
Vision language model, confidence estimation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_MoPqOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

DDS uses changes in localization predictions between consecutive epochs to weight evidential regularization and localization residuals. It retains single-pass inference and improves uncertainty–error alignment across three backbones, two grounding datasets, and zero-adaptation evaluation on Ref-Adv-s.

Critical strengths
  1. The method is simple and requires only a previous-epoch prediction buffer during training, with no additional inference sampling or models.

  2. Uncertainty improvements are consistent across backbones and datasets while largely preserving localization accuracy. The zero-adaptation OOD results provide additional evidence of usefulness.

Critical weaknesses
  1. Novelty and attribution are limited. Equation (6) dynamically weights the existing evidential regression regularizer, while Equation (8) adds a weighted L1 loss. The ablation enables both terms together, without constant-weight or shuffled-dynamics controls. The gains could therefore reflect stronger regularization or additional localization supervision rather than the dynamics signal itself.

  2. The key control is unclear. The no-dynamics row in Table 3 reports Pearson 0.3363, versus 0.5081 for the same backbone’s NIG baseline in Table 1, without explaining the loss or configuration differences. Dropout sampling counts, ensemble size, and variability across independent training runs are also missing.

Questions for authors
  1. Clarification: Does the no-dynamics configuration in Table 3 retain standard evidential regularization? What is its full loss, and why does it differ substantially from the NIG baseline in Table 1?

  2. Additional validation: How do distribution shaping and residual correction perform separately? With matched average loss weights, does actual IEPV outperform constant weights and shuffled IEPV?

  3. Clarification and additional validation: What are the dropout sampling count and ensemble size? Are the main improvements consistent across independent training seeds?

Reviewer_b8zDOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper addresses uncertainty estimation for referring-expression grounding with MLLMs. DDS builds on a Normal-Inverse-Gamma evidential regression head. During training, it caches each sample's previous-epoch box, converts the variation between consecutive epochs into a reliability weight, and uses this weight both to scale the evidence regularizer (by up to 2×) and to weight an additional L1 residual term. Inference remains a single forward pass. Across three LoRA-tuned backbones on RefCOCOg and RefCOCO+, DDS improves the Pearson correlation between uncertainty and 1 − IoU over the NIG baseline by about 0.04–0.06 and lowers AUSE, while localization accuracy stays roughly unchanged. It also slightly improves OOD AUROC when transferring from RefCOCO+ to Ref-Adv-s.

Critical strengths
  1. The method is simple and cheap. It adds only a per-sample buffer during training and has no inference-time cost. The improvement over the matched NIG baseline is consistent in direction across three backbones, two datasets and a zero-adaptation OOD setting.

  2. The problem is practically relevant, since single-pass uncertainty is attractive for multi-billion-parameter grounding models, where ensembles or MC dropout are expensive. The paper is clearly written and openly notes that its localization gains are not uniform.

Critical weaknesses
  1. The novelty is limited, and the mechanism is not explained. With a = 1, the shaping loss is exactly the evidence regularizer of Amini et al. (2020), |y − γ|(2ν + α). DDS therefore amounts to per-sample reweighting of that regularizer by at most 2×, plus an instability-weighted L1 term. Using per-sample training dynamics as a reliability signal is well established (forgetting events, area under the margin, dataset-cartography variability, variance of gradients, and past-epoch self-distillation), yet none of this work is cited or compared against.

    The paper also does not explain why a signal that exists only for training samples should improve uncertainty on test samples. It shows neither how IEPV relates to error nor how IEPV evolves over the 20 LoRA epochs; the signal may vanish once the training set is fit. The two losses are never ablated separately, even though the L1 term is by itself a form of hard-example reweighting. In addition, normalizing IEPV by absolute box coordinates biases it by box position.

  2. The empirical evidence is inconsistent and thin. The "no dynamics" row in Table 3 (Pearson 0.336, AUSE 0.130) differs substantially from the NIG baseline in Table 1 for the same backbone and data (0.508, 0.112), although the two should be the same model. As a result, the ablation gain is about five times the main-table gain. Results come from single runs without standard deviations, several localization differences are 0.1–0.3 points, and the RefCOCOg evaluation split is not stated even though hyperparameters were selected on the validation set. Uncertainty is measured only by Pearson and AUSE, with no calibration metric (NLL, interval coverage) and no risk–coverage analysis. The baselines are limited to MC dropout, ensembles of unspecified size and vanilla NIG; they omit heteroscedastic Gaussian or Laplace likelihoods, IoU-prediction heads, Bayesian LoRA, and the MLLM's own token-level confidence.

  3. The introduction and related work emphasize video temporal grounding, but all experiments use images. LFM2.5-VL's accuracy of about 63% on RefCOCOg versus 92% on RefCOCO+ is not explained. The hyperparameter sweep covers only one backbone.

Questions for authors
  1. Consistency and rigor of the results. Why does the "no dynamics" configuration in Table 3 differ so much from the NIG baseline in Table 1, and which numbers correspond to plain NIG with the standard evidence regularizer? Which RefCOCOg split are the Table 1 results reported on, and which split was used for hyperparameter selection? Can you report mean ± std over at least three seeds? Could you also explain the large accuracy gap of LFM2.5-VL between RefCOCOg and RefCOCO+?

  2. Mechanism and novelty. How does the distribution of IEPV evolve over the training epochs, and how does it correlate with final training and validation error? Could you ablate the shaping loss and the residual-correction loss separately, and try normalizing IEPV by box size rather than by coordinate magnitude? How does DDS compare when the per-sample weight comes from existing training-dynamics signals (forgetting counts, AUM, cartography variability), and how does it compare with self-distillation methods that use past-epoch predictions?

  3. Evaluation of uncertainty. Could you add calibration metrics (NLL, coverage of predictive intervals) and risk–coverage / AURC curves for selective prediction? Could you also compare against heteroscedastic Gaussian/Laplace regression, an IoU-prediction head, Bayesian or Laplace LoRA, and MLLM token-level confidence, and state the ensemble size and MC-dropout settings? Finally, please either add a video temporal-grounding experiment or restrict the claims to image grounding.

Reviewer_sp9COpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper uses changes in bounding-box predictions between consecutive training epochs to adjust an evidential regularizer and an auxiliary regression loss. It evaluates the resulting uncertainty estimates on visual grounding tasks using several VLM backbones.

Critical strengths

The method is simple and preserves single-pass inference. The paper reports improvements in uncertainty–error alignment across multiple backbones.

Critical weaknesses

I found the connection between the task motivation and the method unclear. The paper motivates the approach through ambiguity in visual grounding, but the proposed weighting mechanism is not specific to grounding or VLMs. Similar ideas could potentially apply to other prediction tasks, such as segmentation. The current presentation does not clearly explain what makes this a grounding-specific contribution or establish its broader usefulness as an uncertainty estimation method.

The ablation also does not isolate the value of the proposed dynamics signal. Distribution shaping and the additional regression loss are enabled together, making it difficult to identify the source of the improvement. Comparisons with stronger constant regularization or weighting based on the current prediction error would help establish whether previous-epoch predictions provide additional useful information.

Finally, the baseline results in Tables 1 and 3 need clarification. For LFM2.5-VL on RefCOCOg, the evidential baseline has a Pearson correlation of 0.5081 in Table 1, while the dynamics-disabled configuration has 0.3363 in Table 3.

Questions for authors
  • What explains the difference between the baseline results in Tables 1 and 3?
  • What are the separate effects of distribution shaping and the auxiliary regression loss?
  • Does previous-epoch variation improve over weighting based only on the current prediction error?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.