acceptodds
Under review as a conference paper at ICLR 2027

Was It Never Shown, or Never Seen? Attributing Failure in Observation-Limited 3D Vision-Language Tasks

Abstract

Vision-language systems for volumetric scans, dense 3D scenes, and long videos must construct a budgeted observation before inference. When a prediction fails, aggregate accuracy cannot reveal whether evidence was omitted, the model failed despite adequate evidence, or a useful observation was not selected. We introduce a matched-counterfactual framework for attributing these failures to input-stage (L1), model (L2), and observation-selection (L3) bottlenecks under a declared reference protocol. Cross-domain analysis reveals that observation vulnerability increases sharply under tight budgets and disproportionately affects compact targets. Guided by this diagnosis, we propose a Joint Adaptive Observation Policy that jointly allocates a fixed budget across support, sampling density, representation resolution, and modality-specific mapping. Given a query and a charged low-cost preview, the policy selects a feasible observation. Training combines task reward with dense evidence-adequacy rewards while keeping the task model frozen. It improves low-budget CT grounding, temporal video localisation, and 3D visual grounding over default fixed observations by 26.7, 15.2, and 14.8 points, respectively. Reference-recoverable misses decrease substantially, whereas joint system–reference misses change comparatively little, supporting adaptive observation as an intervention on failures originating before task inference.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Failure Attribution, Adaptive Observation, Vision-Language Models, Budgeted Inference, 3D Vision

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_ZeATOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper proposes (i) a matched-counterfactual failure-attribution framework for vision-language systems that must construct a budgeted observation from a raw 3D/temporal input V (volumetric scans, dense 3D scenes, long videos) before task inference, and (ii) a Joint Adaptive Observation Policy that jointly selects support, sampling density, representation resolution, and modality-specific mapping under a hard token/encoder cost budget. The attribution framework partitions failures into three classes relative to a declared reference set R_B(y): L1 (input-stage failure — no adequate observation in the registry), L2 (model failure — both system and reference adequate observations yielded wrong predictions), L3 (observation-selection failure — the policy selected an inadequate observation while an adequate reference existed). A geometry screen G = e(y)/Δ(a) and realized coverage m ≥ k serve as pre-outcome vulnerability diagnostics. The policy is trained with task reward + dense evidence-adequacy rewards (support, sampling, representation) while the task model is kept frozen after an observation-randomized adaptation phase. Experiments on CT lesion grounding, Charades-STA, and ScanRefer report low-budget gains of +26.7 / +15.2 / +14.8 points over fixed observations and standard-budget gains of +3.2 / +1.4 / +2.6, with ablations isolating support, task reward, query-conditioning, and tuple structure, and paired attribution reductions in low-budget miss/reference-hit outcomes from 47.3 → 22.0 (CT), 31.9 → 18.3 (video), 29.3 → 15.7 (ScanRef

Critical strengths
  1. A useful conceptual carve-out of a real problem. Observation-limited 3D/temporal VLM evaluation is genuinely under-studied: most benchmarks assume the model receives an adequate observation, and when it does not, aggregate accuracy conflates "the evidence never reached the model" with "the evidence reached the model and the model failed." Even granting the framework's reference-dependence issues (see Soundness), the L1/L2/L3 partition is a sensible first-cut lens, and the paper's observation that different interventions (change pipeline vs. improve model vs. improve selection) correspond to different failure classes is correct and actionable.
  2. Non-trivial low-budget empirical gains with a frozen task model. The low-budget +26.7/15.2/14.8 improvements are large enough that, even granting the limited baseline comparison, the joint policy is doing something substantive beyond the fixed-observation default and beyond random selection. The paired attribution reductions (eligible miss/reference-hit from 47.3→22.0, 31.9→18.3, 29.3→15.7) are the right diagnostic for a selection-side intervention and tie the policy's gains to the L3 class the framework predicts it should affect.
Critical weaknesses
  1. The paper is two papers compressed into one, and both suffer. The framework (L1/L2/L3 + G + m + reference-counterfactual attribution) is novel and could carry a focused paper on its own, with proper formalization, cross-domain threshold-sensitivity analysis, and a careful treatment of reference-dependence. The policy (joint registry-constrained adaptive observation) is a reasonable method that could carry its own paper if benchmarked against established active-perception methods on each domain. Bundling them forces both halves to be compressed to the point of near-illegibility (see Presentation: 1), prevents the "joint" claim from being properly ablated, and leaves the framework-to-policy connection loose rather than tight. Either half, developed on its own, would be a stronger contribution than the current combined submission.
  2. The absence of comparison to any existing adaptive observation method on these domains is a significant empirical gap. The paper is clearly aware of Vote-MI, Adaptive Focus, LENS, V*, DeepEyes, ACTIVE-o3, and ActiveVLA (all cited in §2) and positions itself in that line. None are benchmarked. For a method paper proposing a Joint Adaptive Observation Policy, the only quantitative comparisons are to Fixed (static), BS (development-selected best-static), Random, and Reference (an oracle that uses target annotations). This leaves the reader unable to tell whether the policy's gains come from (a) joint allocation specifically, or (b) adaptive observation in general — any domain-specific adaptive method would plausibly close much of the gap to Fixed/Random on its own.
Questions for authors
  1. The paper repeatedly uses L2 as the "model failure" class but §5 and §7 both state that L2 is defined relative to the chosen reference set, not relative to all feasible observations. Under what circumstances is the L1/L2/L3 partition stable to changes in the reference set? If I shrink R_B(y) to a single reference, do many previous L2 cases become L1? Please quantify.
  2. Table 4 compares the Policy to Fixed, BS, Random, Reference, and NP. Can you add a comparison to at least one existing adaptive observation method per domain — e.g., Vote-MI for CT slice selection, Adaptive Focus or LENS for Charades-STA, and an existing 3D proposal method for ScanRefer? Without this comparison, the "+26.7 at low budget" headline cannot be attributed to the specific joint-allocation idea the paper proposes.
  3. Table 6 ablates away single factors (Support only, Task reward only, Query-only, Tuple permutation). Can you add a factorized policy that selects support, density, resolution, and mapping independently under the same budget? This is the natural "non-joint" ablation and is the one that most directly tests the paper's titular "joint" claim.
  4. The +3.2 / +1.4 / +2.6 standard-budget gains are small relative to seed variance. Please report three-seed standard deviations and whether these differences survive a paired test across policy seeds. If they do not, the paper's claim is more accurately "the policy helps at tight budgets" rather than "the policy helps across budgets."
  5. Would you consider restructuring the paper around either the attribution framework or the policy, rather than both? As written the framework is under-formalized (one page) and the policy is under-benchmarked (one page); either half, developed properly, would be a stronger paper than the current submission.
  6. The Reference uses target annotations. Table 4 shows Reference reaching 75.9/61.8/55.4 — the oracle upper bounds. The Policy reaches 56.8/46.9/42.6 at low budgets. What fraction of the Reference–Fixed gap does the Policy actually close, and what does the remaining Policy–Reference gap decompose into (L2, uncovered L3, or something else)?
4 comments

Thank you for the detailed review and for recognizing the value of the attribution framework and the low-budget improvements. We respond to the two critical weaknesses first, followed by your six questions in their original order.

Critical Weakness 1: Organization and the relationship between attribution and policy.

“The paper is two papers compressed into one, and both suffer.”

Thank you for your suggestion. The central contribution of this paper is failure attribution and its diagnostic value. The paper addresses one central question: with the task model fixed and without increasing the observation budget, which failures can be recovered by changing the observation, and how many can a practical observation strategy recover? The attribution framework defines how to evaluate this question, the empirical analysis characterizes the conditions under which observation bottlenecks arise, and the adaptive policy provides a concrete intervention to test whether those bottlenecks can be alleviated. These three components therefore serve the same research objective. The policy is a secondary methodological contribution supporting this objective; the attribution framework and its validation constitute the main contribution.

Retaining the policy allows us to test whether the framework’s diagnosis translates into practical improvements, beyond identifying recoverable failures offline. To make this role clearer, we have rewritten the abstract, introduction, and conclusion around failure attribution and its validation. You also note that combining the framework and policy leaves insufficient room for benchmarking and a proper ablation of the “joint” claim. We have addressed these concerns by adding comparisons with existing adaptive methods and a Joint-versus-Factorized ablation, with results presented below.

Critical Weakness 2: Comparison with existing adaptive observation methods.

“The absence of comparison to any existing adaptive observation method on these domains is a significant empirical gap.”

Thank you for this suggestion. To evaluate our policy against existing adaptive approaches, we have added comparisons with Vote-MI for CT, AdaFocus for video, and 3D-SPS-based point selection for ScanRefer. These methods cover representative slice selection, adaptive video-region selection, and language-guided 3D point selection, respectively.

We adapt Vote-MI to CT multi-slice selection and AdaFocus to temporal grounding. For ScanRefer, we adapt the point-selection mechanism of 3D-SPS to construct observations for the frozen task model. These are task-adapted comparisons, rather than results copied from the original papers. We evaluate them using our task metrics: CT target hit rate, Charades-STA R@1 at tIoU 0.5, and ScanRefer [email protected].

Domain / low-budget anchorAdaptive comparatorComparator scoreJoint Policy scorePolicy − comparator (pp)
CT / 8 slicesVote-MI, CT multi-slice adaptation47.356.8 ± 0.3+9.5
Video / 4 framesAdaFocus, temporal-grounding adaptation43.146.9 ± 0.1+3.8
ScanRefer / 1,024 points3D-SPS-based point selection, adapted to the frozen task model40.942.6 ± 0.2+1.7

Joint Policy entries report mean ± standard deviation across three policy seeds. The existing adaptive methods outperform both BS and NP, while Joint Policy achieves further gains of 9.5, 3.8, and 1.7 percentage points in CT, video, and ScanRefer, respectively. This advantage may reflect our policy’s suitability for low-budget observation, as it learns to allocate limited resources while explicitly rewarding the retention of task-relevant evidence.

Question 1: Attribution stability under changes to the reference set.

“Under what circumstances is the L1/L2/L3 partition stable to changes in the reference set? If I shrink R_B(y) to a single reference, do many previous L2 cases become L1? Please quantify.”

Thank you for raising this question. With the system observation and prediction, task model, decoding procedure, and adequacy criteria fixed, the attribution remains stable whenever changes to the reference set preserve two decisions: (1) whether an adequate reference exists, and (2) if one exists, whether the reference selected by the declared rule yields a successful prediction. For a policy failure, absence of an adequate reference gives L1; failure under the selected adequate reference gives L2; and success under that reference gives L3. Thus, the candidate set and even the selected observation may change without changing the label, provided these decisions remain unchanged.

In the original procedure, we assess all budget-feasible candidate observations for each target and select one reference from those that pass the evidence checks using the declared ranking rule. To test the effect of restricting this search, we instead retain just one observation sampled uniformly from the budget-feasible pool and then check whether it is adequate. The results are:

Domain / low-budget anchorOriginal L2 targetsL2 → L1 under a single candidateFraction of original L2Label agreement among Policy-failed targets
CT / 8 slices596223.69%92.3%
Video / 4 frames1,030474.56%94.9%
ScanRefer / 1,024 points3,119922.95%91.4%

Even when the candidate pool is reduced to a single observation, only 2.95%–4.56% of original L2 targets become L1, and 91.4%–94.9% of failure labels remain unchanged. These results show high attribution stability under the tested single-candidate restriction.

We also examine whether the rule used to choose a reference affects the diagnostic conclusion. From the same pool of adequate observations, we select references using the original ranking, coverage-first ranking, or context-first ranking:

DomainOriginal (adequacy-margin-first)Coverage-firstContext-first
CT25.325.126.9
Video13.614.215.8
ScanRefer13.613.315.3

For each reference-selection rule, we compare Fixed with Policy. Each number shows how many percentage points Policy reduces the rate of errors that the selected Reference can recover. The reductions remain similar across all three rules: 25.1–26.9 points in CT, 13.6–15.8 in video, and 13.3–15.3 in ScanRefer.

Together, these tests show that most individual failure labels survive a substantial reduction in the candidate pool, while the measured benefit of Policy remains consistent when the reference-selection rule changes. This supports both the stability of the attribution and the reliability of its diagnostic conclusion under the tested changes.

Question 2: At least one existing adaptive method per domain.

“Can you add a comparison to at least one existing adaptive observation method per domain — e.g., Vote-MI for CT slice selection, Adaptive Focus or LENS for Charades-STA, and an existing 3D proposal method for ScanRefer?”

Please see our response to Critical Weakness 2 (W2) above. The added comparisons show that Joint Policy exceeds the adapted existing methods by 9.5, 3.8, and 1.7 percentage points in CT, video, and ScanRefer, respectively, showing that Policy also outperforms these adaptive comparators under low budgets.

Question 3: Joint versus factorized selection.

“Can you add a factorized policy that selects support, density, resolution, and mapping independently under the same budget?”

Thank you for this particularly valuable suggestion.

Following your suggestion, we added a Factorized selector that scores support, density, resolution, and modality-specific mapping separately and sums these scores for each complete action. Both selectors receive the same query and preview and use the same training data, task and evidence-adequacy rewards, optimization budget, and frozen task model. Both selectors operate under the same observation-budget constraints.

The low-budget results are reported below as mean ± standard deviation across three training seeds:

SelectorCT / 8 slicesVideo / 4 framesScanRefer / 1,024 points
Factorized52.2 ± 0.943.8 ± 0.540.6 ± 0.6
Joint Policy56.8 ± 0.346.9 ± 0.142.6 ± 0.2
Joint − Factorized (pp)+4.6+3.1+2.0

Factorized selection already achieves substantial improvements over Fixed, showing that query-dependent allocation of individual observation factors is useful. Joint Policy further improves performance in all three domains. These results support an additional benefit from modeling cross-factor interactions under the tested low-budget settings, beyond allowing all four factors to vary.

The comparison also sharpens the interpretation of our existing ablations: Support-only tests whether adjusting the observation range alone is sufficient, whereas Factorized tests whether additive factor scores suffice when all four factors remain adjustable. We have incorporated this comparison into the revised manuscript and clarified this distinction. We appreciate the suggestion, which makes the evidence for joint allocation more direct.

Question 4: Seed variability and standard-budget improvements.

“Please report three-seed standard deviations and whether these differences survive a paired test across policy seeds.”

Thank you for requesting this analysis. We recalculated the standard-budget means and sample standard deviations from the three policy-training runs and corrected the previously reported summaries. The underlying Policy scores are 58.4, 58.2, and 58.1 for CT; 50.1, 50.4, and 50.6 for video; and 53.2, 53.2, and 53.4 for ScanRefer.

Fixed uses the same frozen task model and fixed predictions across policy seeds. We therefore test the three seed-level Policy-minus-Fixed differences against zero using a two-sided one-sample t-test (df = 2).

Domain / standard-budget anchorFixedPolicy, mean ± sample SDPolicy − Fixed (pp)t (df = 2)Two-sided p95% CI for mean gain (pp)
CT / 32 slices55.258.233 ± 0.153+3.03334.3950.000844[2.654, 3.413]
Video / 8 frames48.750.367 ± 0.252+1.66711.4710.007514[1.042, 2.292]
ScanRefer / 40,000 points50.653.267 ± 0.115+2.66740.0000.000624[2.380, 2.954]

At standard budgets, Policy significantly outperforms Fixed in all three domains under this test, including after Bonferroni correction for the three comparisons (p < 0.05/3 = 0.0167).

Question 5: Restructuring around a central contribution.

“Would you consider restructuring the paper around either the attribution framework or the policy, rather than both?”

Yes. As explained in our response to Critical Weakness 1, the revised manuscript centers on failure attribution and its diagnostic value. The empirical analysis characterizes the diagnosed bottlenecks, and the policy provides a concrete intervention that tests their practical recoverability. This organization makes the contribution hierarchy explicit while preserving the experimental link between diagnosis and intervention.

Question 6: Fraction of the Reference–Fixed gap closed and the remaining errors.

“What fraction of the Reference–Fixed gap does the Policy actually close, and what does the remaining Policy–Reference gap decompose into (L2, uncovered L3, or something else)?”

Thank you for this question. At low budgets, Policy closes approximately half of the Reference–Fixed performance gap in each domain:

Domain / low-budget anchorFixedPolicyReferenceFraction of Reference–Fixed gap closedRemaining Reference–Policy gap (pp)
CT / 8 slices30.156.875.958.3%19.1
Video / 4 frames31.746.961.850.5%14.9
ScanRefer / 1,024 points27.842.655.453.6%12.8

The fraction closed is calculated as (Policy − Fixed) / (Reference − Fixed). For example, in CT, the Reference–Fixed gap is 75.9 − 30.1 = 45.8 points, and Policy improves over Fixed by 56.8 − 30.1 = 26.7 points, closing 26.7 / 45.8 = 58.3% of the gap.

On targets with an adequate reference, the remaining gap corresponds to L3—cases where Reference succeeds but Policy fails—minus the reverse cases where Policy succeeds but Reference fails.

Reference uses target annotations for offline selection, whereas Policy has access only to the query and a preview charged to the observation budget. Moreover, in multi-target CT, the target-specific reference outcomes may not be jointly achievable by a single query-level observation. Reference therefore serves as a diagnostic comparison, not a performance upper bound that a deployable policy is required to attain; eliminating the entire gap is not the objective of this work.

To further narrow the gap, future work could explore ranking supervision derived from adequate observations in the training set and improve the allocation of preview information under the same total budget. These directions could translate offline diagnostic findings into observation-selection capabilities available at deployment.

Reviewer_GoPBOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper addresses the problem where VLM systems for 3D and long video modalities often require compression or selection of the input to meet resource constraints. It introduces a failure attribution framework to separate input-stage, model and tool-augmented failures, and makes several observations like small targets are more vulnerable and this is more severe under more limited budgets, sampling density matters, and improving visibility does not necessarily improves performance. It then proposes Joint adaptive Observation Policy, a method to obtain optimal compression strategies. Specifically, it defines an action space over support, density, resolution and mapping, and trains a policy over this action space to compress the input. The paper performs experiments on CT, long video and point cloud modalities, and shows that the proposed policy outperforms alternative baseline selection methods, especially under low budget settings.

Critical strengths
  • The proposed attribution framework and Joint Adaptive Observation Policy are well-motivated and technically sound.
  • Evaluation is performed across three different modalities/tasks/tasks models, and consistent performance gains are observed.
Critical weaknesses
  • Under the standard budget, the proposed method only brings small improvement upon the Fixed baseline. The regime that the proposed method brings the most gain (low) could be too restrictive and less commonly encountered in practice.
Questions for authors
  • It would be helpful to clarify how the low/standard anchors are selected
  • Potentially missing ablation on preview sizes. How was the current size selected and how sensitive the method is to this budget?
  • In FIgure 4(b) the reference performance drops when going from low to standard budget, which could lead to concerns about reference being inaccurate, and consequently leading to inaccurate attributions. It would be helpful to have more clarifications or discussions on the practical impact of this on the stability or reliability of the attribution, the rationale behind the current reference construction and whether there are alternative constructions considered that might mitigate such concerns,.
2 comments

Thank you for recognizing the motivation, technical soundness, and cross-domain evaluation of our work. We address below the practical relevance of the budget settings, the choice of preview allocation, and the implications of reference performance changes for attribution reliability.

Critical Weakness: Practical relevance of low-budget observation.

“Under the standard budget, the proposed method only brings small improvement upon the Fixed baseline. The regime that the proposed method brings the most gain (low) could be too restrictive and less commonly encountered in practice.”

Our central claim is that adaptive observation improves performance under constrained observation budgets while keeping the task model fixed. The motivation for studying this regime comes from practical resource pressures in both training and deployment.

For example, in our CT system, training an approximately 9B-parameter model on approximately 100,000 training examples with 32 slices per example takes approximately 48 hours on eight H100 GPUs, corresponding to approximately 384 GPU-hours. In a resource-constrained clinical deployment scenario, the available hardware may be a single GPU with around 24 GB of memory, such as an RTX 4090. Such a memory-constrained deployment environment makes observation size a practical design consideration.

To test whether the benefit is confined to the most restrictive setting, we added intermediate-budget evaluations:

DomainBudget anchorNPPolicyPolicy − NP
CT8 slices43.256.8+13.6
CT16 slices52.357.1+4.8
CT32 slices58.758.23−0.47
Video4 frames41.046.9+5.9
Video6 frames46.949.2+2.3
Video8 frames50.450.37−0.03
ScanRefer1,024 points36.442.6+6.2
ScanRefer4,096 points44.847.7+2.9
ScanRefer40,000 points53.553.27−0.23

The standard-budget Policy entries above use the corrected three-seed means.

At the intermediate budgets, Policy still improves over NP by 4.8, 2.3, and 2.9 percentage points. These gains show that adaptive observation remains useful beyond the most restrictive setting.

Question 1: Selection of low/standard budget anchors.

“It would be helpful to clarify how the low/standard anchors are selected”

Thank you for asking us to clarify these choices. The standard-budget anchors follow input scales used in representative prior work: 32 CT slices draw on the 32-plane volumetric inputs used by M3D and Med-2E3, eight video frames follow the sampling configuration of Video-LLaVA, and 40,000 ScanRefer points follow its official implementation. For CT, we retain our own slice-selection and rendering procedure.

We selected eight CT slices, four video frames, and 1,024 ScanRefer points as deliberately restrictive settings at the lower end of our evaluation range. The aim was to examine how well the policy preserves task performance when only a small observation budget is available.

To examine performance between these restrictive settings and the standard budgets, we added evaluations with 16 CT slices, six video frames, and 4,096 ScanRefer points, as reported in our response to the weakness above. Policy retains gains of 4.8, 2.3, and 2.9 percentage points over NP, respectively, showing that its benefit extends beyond the most restrictive settings. We appreciate your suggestion and have added the rationale for the budget choices and the intermediate-budget results to the revised manuscript.

Question 2: Choice of preview allocation and sensitivity.

“Potentially missing ablation on preview sizes. How was the current size selected and how sensitive the method is to this budget?”

Thank you for this suggestion. We selected the current preview size through a validation-set ablation conducted before submission and used the best-performing configuration as the default (1×). We now report these ablation results to clarify both the selection and its sensitivity.

Under the low-budget settings, we compare approximately 0.5×, 1×, and 2× the default preview allocation, where 1× uses 256 tokens for CT and video and 64 groups for ScanRefer. The total budget remains fixed, so increasing the preview allocation reduces the resources available for the final observation.

Preview allocationCT hit rateVideo R@1ScanRefer [email protected]
Approximately 0.5×50.644.241.2
Default 1×56.846.942.6
Approximately 2×47.743.838.2
NP, no preview43.241.036.4

The default 1× allocation achieves the highest accuracy in all three domains. The observed sensitivity is consistent with the tight total budgets used in this experiment: a smaller preview may provide insufficient information for selecting an effective observation, while a larger preview leaves fewer resources for the final observation. Under these constraints, allocating either too little or too much to the preview can reduce performance. Nevertheless, all tested allocations outperform NP, showing that the benefit of adaptive observation persists beyond the selected default.

Question 3: Reference performance and attribution reliability.

“In FIgure 4(b) the reference performance drops when going from low to standard budget, which could lead to concerns about reference being inaccurate, and consequently leading to inaccurate attributions.” “…the practical impact of this on the stability or reliability of the attribution, the rationale behind the current reference construction and whether there are alternative constructions considered that might mitigate such concerns…”

Thank you for highlighting the implications for attribution reliability. We address why the reference score decreases and whether alternative reference choices change the diagnostic conclusion.

① Reference selection measures retained evidence, not prediction accuracy. We select the reference based on how well an observation preserves the annotated target evidence, rather than whether it makes the model predict correctly. This allows us to test whether the model succeeds when given sufficient evidence without selecting observations based on its answers. A larger budget can change which reference is selected, so its prediction accuracy need not increase.

② The score decreases after changing the reference, not after increasing the budget alone. With the video evaluation queries, task model, and decoding settings fixed, we obtain:

Evaluation conditionVideo R@1 at tIoU 0.5
Low-budget reference under the low cap61.8
Identical observation replayed under the standard cap61.8
Reference reselected under the standard cap52.5

All 3,720 selected low-budget references remain feasible under the standard cap (100%). Keeping the same observation preserves the score; reselection produces the decrease.

③ Policy consistently reduces recoverable errors across alternative reference-selection rules. These rules choose a reference from the same budget-feasible pool of observations that pass the evidence checks. We compare the original ranking, which prioritizes the weakest normalized evidence margin, with coverage-first and context-first rankings, which prioritize target coverage and retained context, respectively.

DomainOriginal (adequacy-margin-first)Coverage-firstContext-first
CT25.325.126.9
Video13.614.215.8
ScanRefer13.613.315.3

For each reference-selection rule, we compare Fixed with Policy. Each number shows how many percentage points Policy reduces the rate of errors that the selected Reference can recover. The similar reductions across all three rules support the reliability of the main diagnostic conclusion under the tested alternatives, despite the change in reference accuracy across budgets. We have added these experiments and explanations to the revised manuscript. Thank you for prompting us to clarify this distinction.

Reviewer_ndkqOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies a budget allocation problem and proposes an evaluation methodology separating input (L1), model (L2), and selection (L3) stage failures. Although budget allocation and failure attribution is related, it is unclear why they are presented as a unified framework, raising my concern about the consistency between problem and solution. The experimental results raise concerns about both the reliability of the reference-based failure attribution and the practical benefits of the adaptive policy.

Critical strengths

The defined problem is important, aiming to allocate budgets limited resources to improve task performance.

Critical weaknesses
  1. The whole structure and writing of the paper need significant improvements. It combines 3 research questions without clearly explaining their connection: failure attribution (why does a model fail), empirical analysis (how do observation budgets and target geometry affect the failures), and adaptive budget allocation (how should the method allocate). For example, in introduction, the authors claim that “We study evidence-localisable tasks whose retained visual support can be assessed”, which seems to be an evaluation methodology. However, the it remains quite unclear how it related to budget allocation. Although the authors use the findings from failure analysis to motivate adaptive observation, the L1/L2/L3 attribution framework does not appear to be explicitly incorporated into policy optimization. It remains unclear whether the proposed failure-attribution framework is necessary for developing the adaptive observation policy, or whether these are two largely independent contributions. Currently, the paper attempts to claim both contributions without establishing either one as the clear center of the work.

  2. Table 4 and Figure 4 show that: (1) (Fig 4b) The reference becomes even worse with more budgets. This is counterintuitive because the larger budget preserves all actions feasible under the smaller budget. (2) (Table 4) No-preview (NP, denoting it does not process the preview and can instead spend the entire available budget on its final observation) even slightly outperforms policy at a standard budget on all 3 modalities. The authors fail to demonstrate a meaningful advantage of their learned policy.

  3. Minors: Figure 1 is possibly generated. Incorrect author ordering in the LENS reference: The submission lists: Ce Zhang, Jinxi He, Yaqi Xie, and Katia Sycara. The actual publication lists: Ce Zhang, Jinxi He, Katia Sycara, and Yaqi Xie.

Questions for authors

See Weakness.

3 comments

Thank you for recognizing the importance of improving task performance under limited resource budgets and for your thoughtful review. We appreciate your constructive suggestions on the paper’s organization, reference reliability, and practical value of the policy. Your feedback has helped us clarify the central contribution and strengthen the experimental evidence. We address each concern below.

Critical Weakness 1: Organization and the connection between attribution, analysis, and policy.

“It combines 3 research questions without clearly explaining their connection: failure attribution (why does a model fail), empirical analysis (how do observation budgets and target geometry affect the failures), and adaptive budget allocation (how should the method allocate).”

Thank you for your suggestion. The central contribution of this paper is failure attribution and its diagnostic value. The paper addresses one central question: with the task model fixed and without increasing the observation budget, which failures can be recovered by changing the observation, and how many can a practical observation strategy recover? The attribution framework defines how to evaluate this question, the empirical analysis characterizes the conditions under which observation bottlenecks arise, and the adaptive policy provides a concrete intervention to test whether those bottlenecks can be alleviated. These three components therefore serve the same research objective. The policy is a secondary methodological contribution supporting this objective; the attribution framework and its validation constitute the main contribution.

You ask how our focus on “evidence-localisable tasks whose retained visual support can be assessed” relates to budget allocation, rather than only to evaluation. The connection is that assessing evidence retention also provides feedback for learning how to allocate the budget. For example, CT lesion annotations let us check whether the selected slices and crops include the lesion. These checks support reference construction for failure attribution and contribute to the evidence-adequacy rewards used for policy training. The policy therefore learns to allocate limited resources to observations that preserve relevant evidence.

Regarding the relationship between L1/L2/L3 and the policy, L3 identifies the observation-selection failures that the policy aims to reduce. The policy is trained using task reward and evidence-adequacy reward, while the L1/L2/L3 labels assess which failures it recovers and which remain, rather than serving as direct training targets.

Thank you for raising this concern. We have rewritten the abstract, introduction, and conclusion to clarify the central contribution of failure attribution and its diagnostic value, and to explain how the empirical analysis and policy intervention support this contribution.

Critical Weakness 2(1): Reference performance at larger budgets.

“(1) (Fig 4b) The reference becomes even worse with more budgets. This is counterintuitive because the larger budget preserves all actions feasible under the smaller budget.”

① Why the selected reference can score lower. We select the reference based on how well an observation preserves the annotated target evidence, rather than whether it makes the model predict correctly. A larger budget keeps previous observations feasible but can cause the ranking rule to select a different one. The newly selected observation is therefore not guaranteed to achieve higher task accuracy.

② Does the decrease come from increasing the budget or changing the reference? We fixed the video evaluation queries, task model, and decoding settings, comparing an unchanged observation with a reselected reference:

Evaluation conditionVideo R@1 at tIoU 0.5
Low-budget reference under the low cap61.8
Identical observation replayed under the standard cap61.8
Reference reselected under the standard cap52.5

All 3,720 selected low-budget reference observations remained feasible under the standard cap (100%). This audit covers the selected references, not every feasible action. Thus, raising the cap leaves the score unchanged when the observation is retained; the decrease occurs after reselection.

③ Policy consistently reduces reference-recoverable misses across alternative reference-selection rules. These rules determine which observation is selected as the reference from the same budget-feasible candidate pool that passes the evidence checks. We compare the original ranking, which prioritizes the weakest normalized evidence margin, with coverage-first ranking, which prioritizes target coverage, and context-first ranking, which prioritizes retained context. Remaining ties follow the original criteria.

DomainOriginal (adequacy-margin-first)Coverage-firstContext-first
CT25.325.126.9
Video13.614.215.8
ScanRefer13.613.315.3

For each reference-selection rule, we compare Fixed with Policy. Each number shows how many percentage points Policy reduces the rate of errors that the selected Reference can recover. Increasing the budget leaves the score unchanged when we keep the same reference; the score drops only after we select a different reference. Across all three selection rules, Policy still reduces the errors that Reference can recover by similar amounts.

Critical Weakness 2(2): Practical advantage over NP.

“(2) (Table 4) No-preview (NP, denoting it does not process the preview and can instead spend the entire available budget on its final observation) even slightly outperforms policy at a standard budget on all 3 modalities. The authors fail to demonstrate a meaningful advantage of their learned policy.”

Thank you for highlighting this important distinction between budget regimes. The policy's main contribution is improving performance when observation resources are limited. At low budgets, Policy improves over NP by 13.6 points in CT, 5.9 points in video, and 6.2 points in ScanRefer, with preview costs included. These results suggest that adaptive selection is particularly useful when the available budget limits the evidence that a fixed observation can retain.

To provide further evidence beyond the lowest-budget setting, we added intermediate-budget evaluations at 16 CT slices, 6 video frames, and 4,096 ScanRefer points:

Domain and budget anchorsLow: Policy − NPIntermediate: Policy − NPStandard: Policy − NP
CT: 8 / 16 / 32 slices+13.6+4.8−0.47
Video: 4 / 6 / 8 frames+5.9+2.3−0.03
ScanRefer: 1,024 / 4,096 / 40,000 points+6.2+2.9−0.23

Policy retains gains of 4.8, 2.3, and 2.9 points at intermediate budgets, showing that its benefit extends beyond the most restrictive setting. At standard budgets, the additional benefit of adaptive selection may be offset by the preview cost, while NP uses its full budget for the final observation. These results clarify where adaptive selection provides the most value: low and intermediate observation budgets. We have emphasized this scope in the revised manuscript.

Minor comment: Citation correction.

“Incorrect author ordering in the LENS reference: The submission lists: Ce Zhang, Jinxi He, Yaqi Xie, and Katia Sycara. The actual publication lists: Ce Zhang, Jinxi He, Katia Sycara, and Yaqi Xie.”

Thank you for identifying the LENS citation error. We have corrected the author order to Ce Zhang, Jinxi He, Katia Sycara, and Yaqi Xie.

I appreciate the authors' detailed rebuttal and additional experiments, which address some of my empirical concerns. However, my central concerns about the reliability of reference-dependent failure attribution and the loose connection between attribution and policy optimization remain unresolved.

Furthermore, I appreciate that the authors acknowledged the incorrect LENS citation. But it is still unacceptable with such an error in the initial submission.

Overall, I do not find the rebuttal sufficient to change my assessment, and I will maintain my original score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.