- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper identifies textual shortcuts in hierarchical visual recognition with VLMs. During multi-turn training, ground-truth ancestor answers can reveal the correct option and strongly influence subsequent predictions. The proposed framework combines Exchange-Isolated Causal Attention (EICA), which blocks access to previous question–answer exchanges, with a probabilistic hierarchy loss that encourages taxonomic consistency. Experiments show improved recognition accuracy and hierarchical consistency in both in-domain and out-of-domain evaluations.
Critical strengths
-
The motivation is reasonable. Providing ground-truth ancestor answers during training can encourage textual shortcuts and creates a mismatch with independent level-wise testing.
-
The taxonomy analysis and input interventions provide concrete evidence of this issue. The authors also show reduced sensitivity to historical text after training with EICA.
-
The framework is simple. The ablations help distinguish the recognition gains from exchange isolation and the additional improvements in hierarchical consistency from the hierarchy loss.
Critical weaknesses
-
The paper misses a closely related work, StochasT [1], which studies the mismatch between multi-turn training and single-turn evaluation and masks historical dependencies, also on iNat-Plant. The authors should discuss its relationship to EICA and compare the two methods under the same training setting.
-
The gains on Qwen3-VL-8B are modest. HCA improves over HierSFT by only 1.24% and 1.17%, respectively. These results provide limited evidence of a substantial advantage of the stronger backbone in these settings.
[1] Qing et al. StochasT: Learning with Stochastic Turn Depth for Visual Instruction Tuning. 2026. https://arxiv.org/abs/2607.00465.
Questions for authors
-
How does EICA differ from StochasT? Could the authors provide a matched comparison, both with and without the same hierarchy loss?
-
Could the authors analyze why the HCA gains are smaller on Qwen3-VL-8B, particularly for unseen categories? Is the method model-specific?
-
Are the HCA gains mainly driven by improved leaf-level accuracy? The gap between HCA and Acc_leaf does not appear to narrow, particularly on Qwen3-VL-8B.
6 comments
(Part II) Response to Reviewer psCZ
W2 & Q2: HCA gains on Qwen3-VL-8B, particularly for unseen categories, and applicability across backbones.
We respectfully point out that our framework achieves meaningful improvements on Qwen3-VL-8B, particularly when considering the different challenges posed by seen and unseen categories.
- Our method achieves substantial gains on seen categories on Plantae Seen, improving HCA by 2.95 percentage points (10.02% relative improvement) over HierSFT, demonstrating its effectiveness even on the stronger Qwen3-VL-8B backbone.
- Our method generalizes well to unseen categories. Unlike the seen setting, unseen-category recognition requires the model to generalize to categories absent from training, making it substantially more challenging. Nevertheless, our method consistently improves HCA by 1.24 and 1.17 percentage points (5.20% and 4.86% relative gains) on Plantae Unseen and Animalia Unseen, respectively. Our method also consistently improves all three hierarchical overlap metrics (POR, S-POR, and TOR) on both unseen splits, providing complementary evidence of improved hierarchical recognition.
- Our method remains effective across different backbones. We additionally compare the base model, HierSFT, TARA, and our method on LLaVA-1.5-7B across all three evaluation splits, using independent queries at all six taxonomic levels. As reported below, our method improves HCA and the three hierarchical overlap metrics (POR, S-POR, and TOR) over HierSFT by 5.75–11.34 percentage points on Plantae Seen and 3.90–9.69 points on Plantae Unseen. These consistent improvements provide additional evidence that the benefits of our framework extend beyond the Qwen backbones. In the revised manuscript, we have expanded Table 2 to include the LLaVA-1.5-7B results and added the corresponding analysis in Sec. 5.2.
| Model | Dataset | HCA | Acc$_{\mathrm{leaf}}$ | POR | S-POR | TOR | | :--- | :--- | ---: | ---: | ---: | ---: | ---: | | LLaVA-1.5-7B | Plantae Seen | 2.10 | 27.95 | 50.46 | 39.07 | 28.90 | | HierSFT | Plantae Seen | 6.85 | 30.36 | 61.91 | 52.04 | 44.31 | | TARA | Plantae Seen | 5.96 | 30.27 | 63.47 | 50.74 | 43.10 | | Ours | Plantae Seen | 12.60 | 34.96 | 70.30 | 61.19 | 55.65 | | LLaVA-1.5-7B | Plantae Unseen | 2.02 | 29.72 | 49.27 | 37.98 | 27.60 | | HierSFT | Plantae Unseen | 5.46 | 31.24 | 57.68 | 47.76 | 39.16 | | TARA | Plantae Unseen | 4.46 | 31.22 | 59.14 | 46.95 | 38.37 | | Ours | Plantae Unseen | 9.36 | 33.02 | 64.77 | 55.76 | 48.85 | | LLaVA-1.5-7B | Animalia Unseen | 3.29 | 25.81 | 57.74 | 49.36 | 40.68 | | HierSFT | Animalia Unseen | 4.94 | 27.78 | 60.76 | 52.52 | 44.87 | | TARA | Animalia Unseen | 4.21 | 28.02 | 60.50 | 51.92 | 43.78 | | Ours | Animalia Unseen | 5.79 | 30.46 | 61.95 | 52.84 | 45.12 |
(Part III) Response to Reviewer psCZ
Q3: Contribution of leaf-level accuracy to HCA gains and the gap between HCA and leaf-level accuracy.
Thank you for this valuable comment. Our HCA gains primarily stem from correcting ancestor-level predictions rather than leaf-only errors. However, these improvements do not necessarily narrow the gap between HCA and leaf-level accuracy, as newly correct leaf predictions may still have incorrect ancestors. To further quantify these contributions, we conduct an additional paired analysis as follows.
- Paired analysis of HCA gains. We compare HierSFT and our method against their corresponding frozen base models on Qwen2.5-VL-3B and Qwen3-VL-8B, across all three evaluation splits and three training seeds. We categorize path corrections into three types based on the original errors: Leaf Only (incorrect leaf with correct ancestors), Ancestors Only (correct leaf with incorrect ancestors), and Joint (errors in both leaf and ancestors). We report the net contributions of these categories, accounting for both corrected and regressed paths, in percentage points. Their sum equals the overall HCA gain.
- Results and interpretation. Our method achieves leaf-only contributions comparable to HierSFT, while consistently yielding larger combined contributions from ancestor-only and joint corrections across both backbones and all three splits. On Qwen3-VL-8B, these contributions exceed those of HierSFT by 0.99–3.00 percentage points. These results highlight the complementary roles of our components: EICA mitigates historical textual dependencies while preserving shared visual information, whereas the probabilistic hierarchy loss explicitly models taxonomic parent–child relationships by coupling adjacent-level prediction distributions.
| Backbone | Dataset | Method | Leaf Only | Ancestors Only | Joint | ΔHCA | |:---|:---|:---|---:|:---:|---:|---:| | Qwen2.5-VL-3B | Plantae Seen | HierSFT | +0.85 | +7.72 | +4.48 | +13.05 | | | | Ours | +0.85 | +9.13 | +7.80 | +17.78 | | | Plantae Unseen | HierSFT | +0.61 | +5.47 | +2.83 | +8.91 | | | | Ours | +0.87 | +6.41 | +5.59 | +12.87 | | | Animalia Unseen | HierSFT | +1.56 | +1.84 | +1.72 | +5.11 | | | | Ours | +1.76 | +2.73 | +2.53 | +7.02 | | Qwen3-VL-8B | Plantae Seen | HierSFT | +1.33 | +6.04 | +3.23 | +10.61 | | | | Ours | +1.28 | +6.55 | +5.72 | +13.55 | | | Plantae Unseen | HierSFT | +1.65 | +4.05 | +1.97 | +7.67 | | | | Ours | +1.52 | +3.75 | +3.64 | +8.91 | | | Animalia Unseen | HierSFT | +1.34 | +1.84 | +1.29 | +4.47 | | | | Ours | +1.52 | +2.12 | +2.00 | +5.64 |
(Part I) Response to Reviewer psCZ
We sincerely thank you for your valuable feedback. We detail our response below point by point. Any modifications made to the paper are highlighted in blue for your convenience. Please kindly let us know if you have any further concerns.
W1 & Q1: Differences between EICA and StochasT and matched comparisons with and without the same hierarchy loss.