acceptodds
Withdrawn from ICLR 2027

PHANERO: Partial-occlusion Hidden-state Alignment with Neural-dimension Evidence for Recovering Objects

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

A system that already runs a vision-language model (VLM) for captioning or planning can ask it for a bounding box instead of serving a separate detector. Occlusion is what breaks that arrangement: boxes emitted as text become unreliable exactly when the object is partly hidden, and the standard repairs — prompting, supplied geometry, fine-tuning — all act before deployment. PHANERO asks what can be repaired during the decode itself. Occlusion displaces the decoder's final hidden state, and part of that displacement is linearly recoverable. From clean and occluded decodes of the image we fit one map back to the clean state and apply it through a forward hook where coordinates become tokens, keeping the unrepaired boxes alongside. Nothing is trained and no gradient is taken; the map is fit once per model on COCO under synthetic occlusion, then used unchanged on KITTI and HOOT under natural occlusion, where it raises recall at IoU () in all nine modeldomain cells and outscores published pixel-, attention- and visual-token-space repairs in nine of eighteen paired tests, losing one. Keeping the unrepaired boxes means any second decode collects part of that gain, so we pre-registered the test that separates them: the same union built from a norm-matched random write and from a resampled decode. The learned correction clears the stronger null on all three models ( to ; of cells, missing KITTI on two models and Qwen3-VL HOOT), on the weakest model partly as extra boxes. Rescaled to its edit size, maps without the pairing or the cross-dimensional terms fall short of it in of localization tests. What this buys is narrow and cheap. A stock open-vocabulary detector with a tuned threshold still beats every cell we repair; the claim is only that a VLM already in service can localize better under occlusion for one extra decode — a plain query ( by total time) with both decodes batched.

Primary area:
foundation or frontier models, including LLMs
Keywords:
occluded object detection, vision-language models, hidden-state alignment, activation steering, visual grounding

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_W7B6OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

PHANERO is a test time correction method for VLM object localization / referring object detection, especially for partly occluded target objects. PHANERO fits a transform on the frozen LLM decoder's final hidden state from occluded objects to clean ones, using synthetic occlusion in COCO. At inference, the transform corrects the hidden state, and the VLM outputs bounding box as text to recover the localization. PHANERO is evaluated with 3 different VLMs, on COCO as well as KITTI and HOOT, demonstrating improved localization recall.

Critical strengths

S1. The method itself is well motivated and easy to understand. The motivation is to recover the "clean" representation for the occluded objects in the VLM feature space, which is an interesting and under-explored topic.

S2. The method is cheap and flexible as a training-free plugin, which is supported by the experimental results generalizing to different backbones and unseen datasets.

Critical weaknesses

W1. Potentially false or overstated claims, given the existing experiments:

  • W1.1 A central claim is that PHANERO recovers the clean final state. This is neither well-defined nor explicitly measured.
  • W1.2 Related to W1.1, despite the central claim being to recover the occluded objects, the corrected objects as shown in Figs 3-5 do not appear to be heavily occluded.
  • W1.3 The authors claim that the gain does not come from "just add boxes". However the FP-AP results suggest that under a precision aware metric, PHANERO underperforms the baseline, which contradicts this claim.

W2. Flaws in experimental design:

  • W2.1 Baseline comparison is not sufficient. The two null baselines are not very informative and in my opinion should go to the appendix, while more reviewed methods in appendix Table 8 should be reproduced and added to the main tables.
  • W2.2 Possibly weakened main competitor: Localization Heads shows almost no effect, which is clearly different from what their original paper reports, what's the reason for this?
  • W2.3 Artificial occlusion is used to estimate the transform, for which more details and analysis should be provided.

W3. Presentation is poor, the writing style is neither professional nor scientific, making it extremely difficult to follow.

Questions for authors

Please address the critical weaknesses.

Reviewer_rF4bOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

PHANERO proposes to fit a full affine ridge-regression map between final decoder states obtained from clean and synthetically occluded versions of the same image. During the inference, the map modifies the states immediately before the language-model head. The system combines the modified decode’s boxes with selected boxes from the original decode. Experiments are done on both synthetic occlusion and natral occlusion.

Critical strengths
  1. Synthetic-to-natural transfer is interesting in such a task of improving localization in VLM.
  2. The paper has provided multiple baselines decodes. Indeed, the resampling control, norm-matched random intervention, paired versus unpaired maps, and edit-size matching are valuable.
Critical weaknesses
  1. Table 6 reports lower false-positive-aware AP than the stronger null in eight of nine settings. As a result, it only establish recall-oriented candidate generation, but not really a general precision-aware advantage over resampling. Would a report in AUROC be feasible?
  2. The intervention is immediately before a linear LM head. The intervention is therefore functionally equivalent to an adapted output projection and bias, before any unchanged downstream logit processing. I believe there are prior works that has worked on learned transformations of frozen-model activations as a general intervention framework.
  3. Overlap between predictions does not establish correctness against ground truth and the suppression rule does not guarantee this. Take G=(0,0,10,10), original B=(2,0,12,10), and repaired P=(4,0,14,10), they have an IoU(B,G)=2/3, IoU(B,P)=2/3, IoU(P,G)=3/7. The rule discards the correct original prediction because it overlaps the repaired prediction, while the retained prediction fails the evaluation threshold.
  4. Formatting seems odd. Fig. 4 has missing subfigures. Table 3 and Figure 6 have small captions but very large main table/figure.

minor:

  • The “nothing is trained” phrase is somewhat not accurate. Ridge regression learns parameters from data.
Questions for authors
  1. Table 7 mentions that images associated with 176 of 751 COCO test questions were seen during fitting. Is this a potential leakage?
Reviewer_4quwOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper suggests learning a mapping for internal representation of VLM from synthetically occluded to clean pairs of same image once at inference and then apply the same mapping for natually occluded images so improve the recall for occulded objects. However, instead of solely relying on corrected boxes they merge it with original boxes. Empirically, they show based on 3 models and 3 datasets that recall is improved in 6 out of 9 combination compared to other methods while only increasing the cost by 1.05 on median per query.

Critical strengths
  1. Cost-efficient approach to increase recall at inference with minimum additional computational cost.
  2. Comprehensive experiments including 3 datasets and 3 models.
  3. Well-designed ablation study to verify the validty of their choices (paired full map) and validate effectiveness under different scenario (w/o mergin, IOU threshold, occlusion intensity)
  4. Reproduciblity of paper due to release of code.
Critical weaknesses
  1. Modest methodological novelty. The main contribution is applying a map learned from paired clean and occluded states to the final language-decoder representation immediately before the VLM predicts coordinate tokens. The underlying ideas have precedents in feature completion for occlusion (FeatComp++), paired clean–corrupted feature alignment in frozen VLMs (Manifold-Adversarial Adapters), and inference-time editing of VLM representations (ControlMLLM).

  2. Gains depend on the metric and the severity of occlusion. In Table 6, PHANERO’s FP-AP is numerically lower than that of the stronger recall control in eight of nine settings. This limits the evidence for an improvement in overall detection quality. The benefit also weakens under severe occlusion. Above 60% occlusion, Qwen3-VL and Gemma4 show no recall advantage over resampling.

Questions for authors
  1. Does PHANERO still find more occluded objects than resampling when both methods have the same number of false positives per image? I’m especially curious about Cosmos3-Nano. It has the biggest recall gains in Table 6, but it also produces more boxes and scores lower on FP-AP.

  2. How much does it matter that the correction is applied at the final decoder layer? Could you fit the same ridge map on the visual tokens after the projector and compare the two on the same queries, using the same box-merging rule? I’d like to see whether editing the decoder gives a clear advantage.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.