- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The authors propose a diagnostic analysis framework that decomposes the repair process into local corrective evidence, privileged task gain, and inference-time task gain, and systematically evaluate it through matched intervention experiments across multiple nuScenes occupancy models and a CARLA simulation environment. Experimental results demonstrate its effectiveness.
Critical strengths
The research question is novel. It decomposes limited repair gains into three tiers—local evidence, privileged gains, and inference-time gains—and highlights that a small final gain does not mean the method is ineffective.
The controls and reporting are relatively standardized. Total intervention strength is precisely matched, with Uniform and Shuffled controls, scene-cluster-level confidence intervals, and numerical checks using FP64 and finite differences. Non-significant results are also faithfully reported as unestablished, avoiding any forced positive conclusions.
Critical weaknesses
The core question is insufficiently answered, and the main conclusions approach common sense. Although the paper discusses repair limitations, it merely lists phenomena such as local evidence not equaling finite-step gains. Since the privileged allocation uses ground truth from the same record, its superiority over inference-time allocation is expected, and the paper evaluates no actual repair methods, providing little guidance for improvement.
The negative-gain conclusion suffers from insufficient attribution, as it stems from a small-scale learning policy. In PanoOcc, the negative effect comes from a specific policy trained on 70 development scenes, while alternative allocation policies under the same direction yield positive gains, suggesting poor policy training rather than a general rule.
The experimental setup is heterogeneous with small sample sizes and tiny effect sizes, raising doubts about comparability and robustness. Intervention contracts vary across models, overall gains are mostly within fractions of a percentage point, and some subsets rely on very few scenes or near-failing baselines.
The text is overly burdened with jargon and has poor readability. The extensive use of abbreviations, qualifiers, and hidden key details in the appendices makes the core findings difficult to distill and evaluate.
Questions for authors
How sensitive are the observed negative inference-time task gains in PanoOcc to the specific small-scale training setup of the policy, and would alternative optimization strategies or architectures alter the conclusion that privileged task gain does not determine inference-time gain?
2 comments
Dear Reviewer,
We have uploaded a revised manuscript and provided additional clarification on the statistical uncertainty and evaluation protocols in the main discussion.
As the discussion period is approaching its end, we would appreciate any further feedback on our rebuttal and revisions, particularly if any of your concerns remain unresolved.
Thank you for your time and consideration.
We thank the reviewer for their careful reading and constructive feedback. Below, we address the concerns and questions raised, providing additional evidence and clarification of our findings.
W1
Our main claim is not that privileged allocation outperforms inference-time policies, as it can be predicted. Instead, we use privileged information to establish whether a beneficial correction is possible under the tested conditions, and then examine how much of this gain an inference-time policy achieves.
Our experiments go beyond observing this gap. In the original SparseOcc experiment, replacing the inference-time ranking with the privileged ranking increased the task gain from +0.0361 to +0.5480 pp. In an additional PanoOcc experiment, changing only the allocation ranking improved mIoU by 1.7396 pp, even though the first-order advantage decreased by 0.0134 while the finite-loss advantage increased by 0.0522.
These findings help answer our central question: repair performance depends on both the availability of useful corrections and how effectively a policy applies them. Our framework uses controlled comparisons to identify which part of the repair process deserves further investigation and to guide subsequent policy improvements.
W2 & Q
We clarify that the 70 development scenes refer to the original evaluation selection, not policy training. The allocator was trained on 2,048 records covering 553 GT-identified scenes. The new comparison uses 1,221 development records from 66 scenes and three seeds.
To test whether the result depends on training choices, we compared two ways of training the same policy. One learns from local gradient scores (Gradient-trained), while the other minimises the prediction loss after applying corrections (Loss-trained). We tested both using AdamW and SGD with momentum, with two scoring heads.
The original training method reduces mIoU in all four settings, while training on the loss after correction consistently improves it. Therefore, changing the optimizer or scoring head does not change our main finding: the gain achieved at inference time depends on how the policy is trained, even when the available correction remains the same.
We also extended the original Utility policy from three to six epochs for one seed, obtaining a change of +0.0227 pp (95% CI [−0.0089, +0.0554]). This provides an additional check on training duration.
Regarding evaluation scale, our comparisons use matched controls within each model. We report the sample counts and uncertainty separately for each setting rather than combining results across different models. We will make these details and the magnitude of the observed effects clearer in the revised manuscript.
W3
We report results within each model using matched controls rather than combining results across different settings. FlashOCC covers 150 validation scenes, while the new PanoOcc evaluation includes 1,221 records from 66 development scenes. The two external datasets contain 815 and 1,812 frames, respectively, each from one driving sequence. We account for this temporal dependence when estimating uncertainty.
We also report the size of the observed improvements. In PanoOcc, the revised policy improves mIoU by +0.5735 pp over Degraded (95% CI [0.3616, 0.8208]), recovering 16.74% of the matched degradation gap. In contrast, the negative inference-time effects in KITTI-360 are very small, approximately 0.001–0.003 pp. We use these results to study the separation between available repair gains and actual policy performance, rather than claiming substantial practical degradation.
We also corrected the FB-OCC task-metric evaluation. The aligned Uniform mIoU is 49.1146% on the original 190 records, rather than the previously reported 3.2039% obtained from mismatched coordinates.
W4
We will submit a revised manuscript during the rebuttal period to improve clarity and readability. Specifically, we will move the key experimental results from the appendix to the main text, including absolute mIoU, the PanoOcc policy comparisons, and external validation. We will also simplify the presentation of our three research questions, reduce unnecessary jargon and abbreviations, and clarify the main findings and their implications for repair design. Detailed experimental settings and additional results will remain in the appendix.