- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper proposes (i) a matched-counterfactual failure-attribution framework for vision-language systems that must construct a budgeted observation from a raw 3D/temporal input V (volumetric scans, dense 3D scenes, long videos) before task inference, and (ii) a Joint Adaptive Observation Policy that jointly selects support, sampling density, representation resolution, and modality-specific mapping under a hard token/encoder cost budget. The attribution framework partitions failures into three classes relative to a declared reference set R_B(y): L1 (input-stage failure — no adequate observation in the registry), L2 (model failure — both system and reference adequate observations yielded wrong predictions), L3 (observation-selection failure — the policy selected an inadequate observation while an adequate reference existed). A geometry screen G = e(y)/Δ(a) and realized coverage m ≥ k serve as pre-outcome vulnerability diagnostics. The policy is trained with task reward + dense evidence-adequacy rewards (support, sampling, representation) while the task model is kept frozen after an observation-randomized adaptation phase. Experiments on CT lesion grounding, Charades-STA, and ScanRefer report low-budget gains of +26.7 / +15.2 / +14.8 points over fixed observations and standard-budget gains of +3.2 / +1.4 / +2.6, with ablations isolating support, task reward, query-conditioning, and tuple structure, and paired attribution reductions in low-budget miss/reference-hit outcomes from 47.3 → 22.0 (CT), 31.9 → 18.3 (video), 29.3 → 15.7 (ScanRef
Critical strengths
- A useful conceptual carve-out of a real problem. Observation-limited 3D/temporal VLM evaluation is genuinely under-studied: most benchmarks assume the model receives an adequate observation, and when it does not, aggregate accuracy conflates "the evidence never reached the model" with "the evidence reached the model and the model failed." Even granting the framework's reference-dependence issues (see Soundness), the L1/L2/L3 partition is a sensible first-cut lens, and the paper's observation that different interventions (change pipeline vs. improve model vs. improve selection) correspond to different failure classes is correct and actionable.
- Non-trivial low-budget empirical gains with a frozen task model. The low-budget +26.7/15.2/14.8 improvements are large enough that, even granting the limited baseline comparison, the joint policy is doing something substantive beyond the fixed-observation default and beyond random selection. The paired attribution reductions (eligible miss/reference-hit from 47.3→22.0, 31.9→18.3, 29.3→15.7) are the right diagnostic for a selection-side intervention and tie the policy's gains to the L3 class the framework predicts it should affect.
Critical weaknesses
- The paper is two papers compressed into one, and both suffer. The framework (L1/L2/L3 + G + m + reference-counterfactual attribution) is novel and could carry a focused paper on its own, with proper formalization, cross-domain threshold-sensitivity analysis, and a careful treatment of reference-dependence. The policy (joint registry-constrained adaptive observation) is a reasonable method that could carry its own paper if benchmarked against established active-perception methods on each domain. Bundling them forces both halves to be compressed to the point of near-illegibility (see Presentation: 1), prevents the "joint" claim from being properly ablated, and leaves the framework-to-policy connection loose rather than tight. Either half, developed on its own, would be a stronger contribution than the current combined submission.
- The absence of comparison to any existing adaptive observation method on these domains is a significant empirical gap. The paper is clearly aware of Vote-MI, Adaptive Focus, LENS, V*, DeepEyes, ACTIVE-o3, and ActiveVLA (all cited in §2) and positions itself in that line. None are benchmarked. For a method paper proposing a Joint Adaptive Observation Policy, the only quantitative comparisons are to Fixed (static), BS (development-selected best-static), Random, and Reference (an oracle that uses target annotations). This leaves the reader unable to tell whether the policy's gains come from (a) joint allocation specifically, or (b) adaptive observation in general — any domain-specific adaptive method would plausibly close much of the gap to Fixed/Random on its own.
Questions for authors
- The paper repeatedly uses L2 as the "model failure" class but §5 and §7 both state that L2 is defined relative to the chosen reference set, not relative to all feasible observations. Under what circumstances is the L1/L2/L3 partition stable to changes in the reference set? If I shrink R_B(y) to a single reference, do many previous L2 cases become L1? Please quantify.
- Table 4 compares the Policy to Fixed, BS, Random, Reference, and NP. Can you add a comparison to at least one existing adaptive observation method per domain — e.g., Vote-MI for CT slice selection, Adaptive Focus or LENS for Charades-STA, and an existing 3D proposal method for ScanRefer? Without this comparison, the "+26.7 at low budget" headline cannot be attributed to the specific joint-allocation idea the paper proposes.
- Table 6 ablates away single factors (Support only, Task reward only, Query-only, Tuple permutation). Can you add a factorized policy that selects support, density, resolution, and mapping independently under the same budget? This is the natural "non-joint" ablation and is the one that most directly tests the paper's titular "joint" claim.
- The +3.2 / +1.4 / +2.6 standard-budget gains are small relative to seed variance. Please report three-seed standard deviations and whether these differences survive a paired test across policy seeds. If they do not, the paper's claim is more accurately "the policy helps at tight budgets" rather than "the policy helps across budgets."
- Would you consider restructuring the paper around either the attribution framework or the policy, rather than both? As written the framework is under-formalized (one page) and the policy is under-benchmarked (one page); either half, developed properly, would be a stronger paper than the current submission.
- The Reference uses target annotations. Table 4 shows Reference reaching 75.9/61.8/55.4 — the oracle upper bounds. The Policy reaches 56.8/46.9/42.6 at low budgets. What fraction of the Reference–Fixed gap does the Policy actually close, and what does the remaining Policy–Reference gap decompose into (L2, uncovered L3, or something else)?
4 comments
Question 1: Attribution stability under changes to the reference set.
“Under what circumstances is the L1/L2/L3 partition stable to changes in the reference set? If I shrink R_B(y) to a single reference, do many previous L2 cases become L1? Please quantify.”
Thank you for raising this question. With the system observation and prediction, task model, decoding procedure, and adequacy criteria fixed, the attribution remains stable whenever changes to the reference set preserve two decisions: (1) whether an adequate reference exists, and (2) if one exists, whether the reference selected by the declared rule yields a successful prediction. For a policy failure, absence of an adequate reference gives L1; failure under the selected adequate reference gives L2; and success under that reference gives L3. Thus, the candidate set and even the selected observation may change without changing the label, provided these decisions remain unchanged.
In the original procedure, we assess all budget-feasible candidate observations for each target and select one reference from those that pass the evidence checks using the declared ranking rule. To test the effect of restricting this search, we instead retain just one observation sampled uniformly from the budget-feasible pool and then check whether it is adequate. The results are:
| Domain / low-budget anchor | Original L2 targets | L2 → L1 under a single candidate | Fraction of original L2 | Label agreement among Policy-failed targets |
|---|---|---|---|---|
| CT / 8 slices | 596 | 22 | 3.69% | 92.3% |
| Video / 4 frames | 1,030 | 47 | 4.56% | 94.9% |
| ScanRefer / 1,024 points | 3,119 | 92 | 2.95% | 91.4% |
Even when the candidate pool is reduced to a single observation, only 2.95%–4.56% of original L2 targets become L1, and 91.4%–94.9% of failure labels remain unchanged. These results show high attribution stability under the tested single-candidate restriction.
We also examine whether the rule used to choose a reference affects the diagnostic conclusion. From the same pool of adequate observations, we select references using the original ranking, coverage-first ranking, or context-first ranking:
| Domain | Original (adequacy-margin-first) | Coverage-first | Context-first |
|---|---|---|---|
| CT | 25.3 | 25.1 | 26.9 |
| Video | 13.6 | 14.2 | 15.8 |
| ScanRefer | 13.6 | 13.3 | 15.3 |
For each reference-selection rule, we compare Fixed with Policy. Each number shows how many percentage points Policy reduces the rate of errors that the selected Reference can recover. The reductions remain similar across all three rules: 25.1–26.9 points in CT, 13.6–15.8 in video, and 13.3–15.3 in ScanRefer.
Together, these tests show that most individual failure labels survive a substantial reduction in the candidate pool, while the measured benefit of Policy remains consistent when the reference-selection rule changes. This supports both the stability of the attribution and the reliability of its diagnostic conclusion under the tested changes.
Question 2: At least one existing adaptive method per domain.
“Can you add a comparison to at least one existing adaptive observation method per domain — e.g., Vote-MI for CT slice selection, Adaptive Focus or LENS for Charades-STA, and an existing 3D proposal method for ScanRefer?”
Please see our response to Critical Weakness 2 (W2) above. The added comparisons show that Joint Policy exceeds the adapted existing methods by 9.5, 3.8, and 1.7 percentage points in CT, video, and ScanRefer, respectively, showing that Policy also outperforms these adaptive comparators under low budgets.
Question 3: Joint versus factorized selection.
“Can you add a factorized policy that selects support, density, resolution, and mapping independently under the same budget?”
Thank you for this particularly valuable suggestion.
Following your suggestion, we added a Factorized selector that scores support, density, resolution, and modality-specific mapping separately and sums these scores for each complete action. Both selectors receive the same query and preview and use the same training data, task and evidence-adequacy rewards, optimization budget, and frozen task model. Both selectors operate under the same observation-budget constraints.
The low-budget results are reported below as mean ± standard deviation across three training seeds:
| Selector | CT / 8 slices | Video / 4 frames | ScanRefer / 1,024 points |
|---|---|---|---|
| Factorized | 52.2 ± 0.9 | 43.8 ± 0.5 | 40.6 ± 0.6 |
| Joint Policy | 56.8 ± 0.3 | 46.9 ± 0.1 | 42.6 ± 0.2 |
| Joint − Factorized (pp) | +4.6 | +3.1 | +2.0 |
Factorized selection already achieves substantial improvements over Fixed, showing that query-dependent allocation of individual observation factors is useful. Joint Policy further improves performance in all three domains. These results support an additional benefit from modeling cross-factor interactions under the tested low-budget settings, beyond allowing all four factors to vary.
The comparison also sharpens the interpretation of our existing ablations: Support-only tests whether adjusting the observation range alone is sufficient, whereas Factorized tests whether additive factor scores suffice when all four factors remain adjustable. We have incorporated this comparison into the revised manuscript and clarified this distinction. We appreciate the suggestion, which makes the evidence for joint allocation more direct.
Question 4: Seed variability and standard-budget improvements.
“Please report three-seed standard deviations and whether these differences survive a paired test across policy seeds.”
Thank you for requesting this analysis. We recalculated the standard-budget means and sample standard deviations from the three policy-training runs and corrected the previously reported summaries. The underlying Policy scores are 58.4, 58.2, and 58.1 for CT; 50.1, 50.4, and 50.6 for video; and 53.2, 53.2, and 53.4 for ScanRefer.
Fixed uses the same frozen task model and fixed predictions across policy seeds. We therefore test the three seed-level Policy-minus-Fixed differences against zero using a two-sided one-sample t-test (df = 2).
| Domain / standard-budget anchor | Fixed | Policy, mean ± sample SD | Policy − Fixed (pp) | t (df = 2) | Two-sided p | 95% CI for mean gain (pp) |
|---|---|---|---|---|---|---|
| CT / 32 slices | 55.2 | 58.233 ± 0.153 | +3.033 | 34.395 | 0.000844 | [2.654, 3.413] |
| Video / 8 frames | 48.7 | 50.367 ± 0.252 | +1.667 | 11.471 | 0.007514 | [1.042, 2.292] |
| ScanRefer / 40,000 points | 50.6 | 53.267 ± 0.115 | +2.667 | 40.000 | 0.000624 | [2.380, 2.954] |
At standard budgets, Policy significantly outperforms Fixed in all three domains under this test, including after Bonferroni correction for the three comparisons (p < 0.05/3 = 0.0167).
Question 5: Restructuring around a central contribution.
“Would you consider restructuring the paper around either the attribution framework or the policy, rather than both?”
Yes. As explained in our response to Critical Weakness 1, the revised manuscript centers on failure attribution and its diagnostic value. The empirical analysis characterizes the diagnosed bottlenecks, and the policy provides a concrete intervention that tests their practical recoverability. This organization makes the contribution hierarchy explicit while preserving the experimental link between diagnosis and intervention.
Question 6: Fraction of the Reference–Fixed gap closed and the remaining errors.
“What fraction of the Reference–Fixed gap does the Policy actually close, and what does the remaining Policy–Reference gap decompose into (L2, uncovered L3, or something else)?”
Thank you for this question. At low budgets, Policy closes approximately half of the Reference–Fixed performance gap in each domain:
| Domain / low-budget anchor | Fixed | Policy | Reference | Fraction of Reference–Fixed gap closed | Remaining Reference–Policy gap (pp) |
|---|---|---|---|---|---|
| CT / 8 slices | 30.1 | 56.8 | 75.9 | 58.3% | 19.1 |
| Video / 4 frames | 31.7 | 46.9 | 61.8 | 50.5% | 14.9 |
| ScanRefer / 1,024 points | 27.8 | 42.6 | 55.4 | 53.6% | 12.8 |
The fraction closed is calculated as (Policy − Fixed) / (Reference − Fixed). For example, in CT, the Reference–Fixed gap is 75.9 − 30.1 = 45.8 points, and Policy improves over Fixed by 56.8 − 30.1 = 26.7 points, closing 26.7 / 45.8 = 58.3% of the gap.
On targets with an adequate reference, the remaining gap corresponds to L3—cases where Reference succeeds but Policy fails—minus the reverse cases where Policy succeeds but Reference fails.
Reference uses target annotations for offline selection, whereas Policy has access only to the query and a preview charged to the observation budget. Moreover, in multi-target CT, the target-specific reference outcomes may not be jointly achievable by a single query-level observation. Reference therefore serves as a diagnostic comparison, not a performance upper bound that a deployable policy is required to attain; eliminating the entire gap is not the objective of this work.
To further narrow the gap, future work could explore ranking supervision derived from adequate observations in the training set and improve the allocation of preview information under the same total budget. These directions could translate offline diagnostic findings into observation-selection capabilities available at deployment.
Thank you for the detailed review and for recognizing the value of the attribution framework and the low-budget improvements. We respond to the two critical weaknesses first, followed by your six questions in their original order.
Critical Weakness 1: Organization and the relationship between attribution and policy.
Thank you for your suggestion. The central contribution of this paper is failure attribution and its diagnostic value. The paper addresses one central question: with the task model fixed and without increasing the observation budget, which failures can be recovered by changing the observation, and how many can a practical observation strategy recover? The attribution framework defines how to evaluate this question, the empirical analysis characterizes the conditions under which observation bottlenecks arise, and the adaptive policy provides a concrete intervention to test whether those bottlenecks can be alleviated. These three components therefore serve the same research objective. The policy is a secondary methodological contribution supporting this objective; the attribution framework and its validation constitute the main contribution.
Retaining the policy allows us to test whether the framework’s diagnosis translates into practical improvements, beyond identifying recoverable failures offline. To make this role clearer, we have rewritten the abstract, introduction, and conclusion around failure attribution and its validation. You also note that combining the framework and policy leaves insufficient room for benchmarking and a proper ablation of the “joint” claim. We have addressed these concerns by adding comparisons with existing adaptive methods and a Joint-versus-Factorized ablation, with results presented below.
Critical Weakness 2: Comparison with existing adaptive observation methods.
Thank you for this suggestion. To evaluate our policy against existing adaptive approaches, we have added comparisons with Vote-MI for CT, AdaFocus for video, and 3D-SPS-based point selection for ScanRefer. These methods cover representative slice selection, adaptive video-region selection, and language-guided 3D point selection, respectively.
We adapt Vote-MI to CT multi-slice selection and AdaFocus to temporal grounding. For ScanRefer, we adapt the point-selection mechanism of 3D-SPS to construct observations for the frozen task model. These are task-adapted comparisons, rather than results copied from the original papers. We evaluate them using our task metrics: CT target hit rate, Charades-STA R@1 at tIoU 0.5, and ScanRefer [email protected].
Joint Policy entries report mean ± standard deviation across three policy seeds. The existing adaptive methods outperform both BS and NP, while Joint Policy achieves further gains of 9.5, 3.8, and 1.7 percentage points in CT, video, and ScanRefer, respectively. This advantage may reflect our policy’s suitability for low-budget observation, as it learns to allocate limited resources while explicitly rewarding the retention of task-relevant evidence.