acceptodds
Under review as a conference paper at ICLR 2027

Beyond Textual Shortcuts: A Training Framework for Hierarchical Visual Recognition with Vision-Language Models

Abstract

Hierarchical visual recognition requires vision-language models (VLMs) to make accurate and taxonomically consistent predictions across multiple semantic levels. However, existing hierarchical visual instruction tuning methods expose preceding ground-truth answers during training, potentially encouraging textual shortcuts rather than visually grounded recognition. Through a reference-taxonomy analysis and controlled interventions, we reveal that historical textual context can determine many training answers without visual evidence and that trained models exhibit substantial reliance on preceding taxonomic cues. To address this issue, we propose a decoupled hierarchical training framework that separates hierarchical supervision from historical textual dependencies. Specifically, it combines Exchange-Isolated Causal Attention (EICA), which isolates preceding question–answer exchanges while preserving shared visual information, with a probabilistic hierarchy loss that explicitly encourages taxonomic consistency across prediction levels. Experiments on iNat21-Plant and iNat21-Animal with two generative VLM backbones demonstrate consistent improvements in hierarchical consistency and fine-grained recognition across seen, unseen, and cross-domain settings.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Hierarchical Visual Recognition, Vision-Language Models

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_psCZOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper identifies textual shortcuts in hierarchical visual recognition with VLMs. During multi-turn training, ground-truth ancestor answers can reveal the correct option and strongly influence subsequent predictions. The proposed framework combines Exchange-Isolated Causal Attention (EICA), which blocks access to previous question–answer exchanges, with a probabilistic hierarchy loss that encourages taxonomic consistency. Experiments show improved recognition accuracy and hierarchical consistency in both in-domain and out-of-domain evaluations.

Critical strengths
  1. The motivation is reasonable. Providing ground-truth ancestor answers during training can encourage textual shortcuts and creates a mismatch with independent level-wise testing.

  2. The taxonomy analysis and input interventions provide concrete evidence of this issue. The authors also show reduced sensitivity to historical text after training with EICA.

  3. The framework is simple. The ablations help distinguish the recognition gains from exchange isolation and the additional improvements in hierarchical consistency from the hierarchy loss.

Critical weaknesses
  1. The paper misses a closely related work, StochasT [1], which studies the mismatch between multi-turn training and single-turn evaluation and masks historical dependencies, also on iNat-Plant. The authors should discuss its relationship to EICA and compare the two methods under the same training setting.

  2. The gains on Qwen3-VL-8B are modest. HCA improves over HierSFT by only 1.24% and 1.17%, respectively. These results provide limited evidence of a substantial advantage of the stronger backbone in these settings.

[1] Qing et al. StochasT: Learning with Stochastic Turn Depth for Visual Instruction Tuning. 2026. https://arxiv.org/abs/2607.00465.

Questions for authors
  1. How does EICA differ from StochasT? Could the authors provide a matched comparison, both with and without the same hierarchy loss?

  2. Could the authors analyze why the HCA gains are smaller on Qwen3-VL-8B, particularly for unseen categories? Is the method model-specific?

  3. Are the HCA gains mainly driven by improved leaf-level accuracy? The gap between HCA and Acc_leaf does not appear to narrow, particularly on Qwen3-VL-8B.

6 comments
AuthorsOpenReview ↗

(Part I) Response to Reviewer psCZ

We sincerely thank you for your valuable feedback. We detail our response below point by point. Any modifications made to the paper are highlighted in blue for your convenience. Please kindly let us know if you have any further concerns.

W1 & Q1: Differences between EICA and StochasT and matched comparisons with and without the same hierarchy loss.

Thank you for bringing the recent work StochasT to our attention. We clarify the differences between StochasT and our method and provide controlled comparisons with and without the same hierarchy loss.

  • Our method differs from StochasT in both research objectives and methodological designs.
    • Research objectives. StochasT targets general visual instruction tuning by adapting to varying historical conversational contexts. In contrast, our work focuses on hierarchical visual recognition, where predictions across taxonomic levels are closely linked through parent–child relationships. We aim to eliminate shortcuts from historical taxonomic answers while preserving hierarchical knowledge.
    • Methodological designs. StochasT stochastically retains historical dependencies, whereas our EICA deterministically eliminates cross-exchange textual dependencies and uses a probabilistic hierarchy loss to preserve parent–child relationships in the prediction space.
  • Following your suggestion, we conduct matched comparisons under identical training settings, both with and without the hierarchy loss.
    • Without the hierarchy loss, our method outperforms StochasT on all five metrics across all three evaluation splits, demonstrating the independent benefits of exchange-level isolation in mitigating historical textual shortcuts.
    • With the same hierarchy loss, our full framework consistently outperforms StochasT + Hierarchy Loss across all metrics and splits, further demonstrating that our improvements cannot be attributed solely to the hierarchy loss and highlighting the complementary benefits of exchange-level isolation and explicit hierarchical modeling.
  • We have added complete results across five metrics and the discussion of StochasT to the revised manuscript (Table 11, Appendix G.6).
    | Method | Hierarchy_Loss | | | Plantae Seen (HCA/Acc${\mathrm{leaf}})∣PlantaeUnseen(HCA/Acc) | Plantae Unseen (HCA/Acc{\mathrm{leaf}})∣AnimaliaUnseen(HCA/Acc) | Animalia Unseen (HCA/Acc_{\mathrm{leaf}}$) | |---|:---:|---|---|---|---|---| | StochasT | w/o | | | 25.69 / 47.05 | 20.81 / 44.79 | 21.88 / 47.77 | | Ours | w/o | | | 26.65 / 49.19 | 21.32 / 46.87 | 22.05 / 48.32 | | StochasT | w/ | | | 27.17 / 46.55 | 22.21 / 44.59 | 22.54 / 47.16 | | Ours | w/ | | | 28.29 / 48.73 | 23.05 / 46.86 | 22.96 / 47.72 |
AuthorsOpenReview ↗

(Part II) Response to Reviewer psCZ

W2 & Q2: HCA gains on Qwen3-VL-8B, particularly for unseen categories, and applicability across backbones.

We respectfully point out that our framework achieves meaningful improvements on Qwen3-VL-8B, particularly when considering the different challenges posed by seen and unseen categories.

  • Our method achieves substantial gains on seen categories on Plantae Seen, improving HCA by 2.95 percentage points (10.02% relative improvement) over HierSFT, demonstrating its effectiveness even on the stronger Qwen3-VL-8B backbone.
  • Our method generalizes well to unseen categories. Unlike the seen setting, unseen-category recognition requires the model to generalize to categories absent from training, making it substantially more challenging. Nevertheless, our method consistently improves HCA by 1.24 and 1.17 percentage points (5.20% and 4.86% relative gains) on Plantae Unseen and Animalia Unseen, respectively. Our method also consistently improves all three hierarchical overlap metrics (POR, S-POR, and TOR) on both unseen splits, providing complementary evidence of improved hierarchical recognition.
  • Our method remains effective across different backbones. We additionally compare the base model, HierSFT, TARA, and our method on LLaVA-1.5-7B across all three evaluation splits, using independent queries at all six taxonomic levels. As reported below, our method improves HCA and the three hierarchical overlap metrics (POR, S-POR, and TOR) over HierSFT by 5.75–11.34 percentage points on Plantae Seen and 3.90–9.69 points on Plantae Unseen. These consistent improvements provide additional evidence that the benefits of our framework extend beyond the Qwen backbones. In the revised manuscript, we have expanded Table 2 to include the LLaVA-1.5-7B results and added the corresponding analysis in Sec. 5.2.
    | Model \qquad | Dataset \qquad | HCA | Acc$_{\mathrm{leaf}}$ | POR | S-POR | TOR | | :--- | :--- | ---: | ---: | ---: | ---: | ---: | | LLaVA-1.5-7B \qquad | Plantae Seen \qquad | 2.10 | 27.95 | 50.46 | 39.07 | 28.90 | | HierSFT \qquad | Plantae Seen \qquad | 6.85 | 30.36 | 61.91 | 52.04 | 44.31 | | TARA \qquad | Plantae Seen \qquad | 5.96 | 30.27 | 63.47 | 50.74 | 43.10 | | Ours \qquad | Plantae Seen \qquad | 12.60 | 34.96 | 70.30 | 61.19 | 55.65 | | LLaVA-1.5-7B \qquad | Plantae Unseen \qquad | 2.02 | 29.72 | 49.27 | 37.98 | 27.60 | | HierSFT \qquad | Plantae Unseen \qquad | 5.46 | 31.24 | 57.68 | 47.76 | 39.16 | | TARA \qquad | Plantae Unseen \qquad | 4.46 | 31.22 | 59.14 | 46.95 | 38.37 | | Ours \qquad | Plantae Unseen \qquad | 9.36 | 33.02 | 64.77 | 55.76 | 48.85 | | LLaVA-1.5-7B \qquad | Animalia Unseen \qquad | 3.29 | 25.81 | 57.74 | 49.36 | 40.68 | | HierSFT \qquad | Animalia Unseen \qquad | 4.94 | 27.78 | 60.76 | 52.52 | 44.87 | | TARA \qquad | Animalia Unseen \qquad | 4.21 | 28.02 | 60.50 | 51.92 | 43.78 | | Ours \qquad | Animalia Unseen \qquad | 5.79 | 30.46 | 61.95 | 52.84 | 45.12 |
AuthorsOpenReview ↗

(Part III) Response to Reviewer psCZ

Q3: Contribution of leaf-level accuracy to HCA gains and the gap between HCA and leaf-level accuracy.

Thank you for this valuable comment. Our HCA gains primarily stem from correcting ancestor-level predictions rather than leaf-only errors. However, these improvements do not necessarily narrow the gap between HCA and leaf-level accuracy, as newly correct leaf predictions may still have incorrect ancestors. To further quantify these contributions, we conduct an additional paired analysis as follows.

  • Paired analysis of HCA gains. We compare HierSFT and our method against their corresponding frozen base models on Qwen2.5-VL-3B and Qwen3-VL-8B, across all three evaluation splits and three training seeds. We categorize path corrections into three types based on the original errors: Leaf Only (incorrect leaf with correct ancestors), Ancestors Only (correct leaf with incorrect ancestors), and Joint (errors in both leaf and ancestors). We report the net contributions of these categories, accounting for both corrected and regressed paths, in percentage points. Their sum equals the overall HCA gain.
  • Results and interpretation. Our method achieves leaf-only contributions comparable to HierSFT, while consistently yielding larger combined contributions from ancestor-only and joint corrections across both backbones and all three splits. On Qwen3-VL-8B, these contributions exceed those of HierSFT by 0.99–3.00 percentage points. These results highlight the complementary roles of our components: EICA mitigates historical textual dependencies while preserving shared visual information, whereas the probabilistic hierarchy loss explicitly models taxonomic parent–child relationships by coupling adjacent-level prediction distributions.
    | Backbone | Dataset | Method | Leaf Only | Ancestors Only | Joint   | ΔHCA | |:---|:---|:---|---:|:---:|---:|---:| | Qwen2.5-VL-3B | Plantae Seen | HierSFT | +0.85 | +7.72 | +4.48   | +13.05 | | | | Ours | +0.85 | +9.13 | +7.80   | +17.78 | | | Plantae Unseen | HierSFT | +0.61 | +5.47 | +2.83   | +8.91 | | | | Ours | +0.87 | +6.41 | +5.59   | +12.87 | | | Animalia Unseen | HierSFT | +1.56 | +1.84 | +1.72   | +5.11 | | | | Ours | +1.76 | +2.73 | +2.53   | +7.02 | | Qwen3-VL-8B | Plantae Seen | HierSFT | +1.33 | +6.04 | +3.23   | +10.61 | | | | Ours | +1.28 | +6.55 | +5.72   | +13.55 | | | Plantae Unseen | HierSFT | +1.65 | +4.05 | +1.97   | +7.67 | | | | Ours | +1.52 | +3.75 | +3.64   | +8.91 | | | Animalia Unseen | HierSFT | +1.34 | +1.84 | +1.29   | +4.47 | | | | Ours | +1.52 | +2.12 | +2.00   | +5.64 |
Reviewer_nW88OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This work focuses on fine-grained visual categorization by leveraging the taxonomic hierarchy of biological species and attempting to classify visual instances at every hierarchical rank.

Critical strengths
  1. The paper is clearly written, and the reported empirical improvements over the chosen baselines are substantial.
Critical weaknesses
  1. Questionable Task Significance: The formulation of hierarchical classification appears to be an artificially constructed task. In practice, the problem can be trivially simplified to standard FGVC (e.g., recognizing specific species or car trims), where higher-level hierarchical attributes can be accurately obtained via deterministic lookups against public knowledge bases (e.g., Wikipedia). As such, explicit multi-level prediction feels largely redundant.
  2. Narrow Domain Scope: The proposed hierarchical setup seems exclusively applicable to biology. Fine-grained classification, however, spans many domains such as vehicles, aircraft, and retail goods, where hierarchical structures are often arbitrary or ill-defined. This suggests that the studied hierarchy is an idiosyncratic quirk of biological datasets rather than a pervasive problem in general FGVC.
  3. Over-emphasis on a Specific Artifact: The "text shortcut" phenomenon extensively discussed in the paper appears to be an artifact specific to the TARA baseline, rather than an intrinsic, general challenge across hierarchical classification. It should not be framed as a general open problem in hierarchical learning.
  4. Insufficient Baseline Comparisons & Missing Competitive Baselines: The empirical evaluation lacks comparisons with recent open-source efforts like Fine-R1. More importantly, the paper omits the most competitive paradigms in FGVC, such as simple supervised fine-tuning (SFT) on CLIP, which often establishes a much stronger baseline. (Note: Performance on unseen categories is of little concern to me, as expecting scalable generalization to concepts never encountered during training is fundamentally unrealistic—humans cannot identify unseen fine-grained classes without prior exposure, let alone models).
  5. Marginal General Task Improvements & Lack of Statistical Rigor: The reported gains on general tasks border on random noise. I recommend testing on stronger open backbones (such as Qwen3-VL), evaluating on comprehensive benchmarks like MMStar, and reporting results with 95% confidence intervals to demonstrate statistical significance.
Questions for authors
  1. Is this framework inherently confined to the biological domain, or does it generalize to other fine-grained domains where standardized taxonomies are absent?
  2. What are the concrete, fundamental advantages of this multi-level prediction approach compared to the straightforward pipeline of standard fine-grained classification followed by a post-hoc taxonomic lookup?
  3. A more comprehensive benchmark comparison is necessary; simple yet strong baselines like CLIP + SFT frequently outperform complex VLM training recipes and must be included for a fair evaluation.
8 comments
AuthorsOpenReview ↗

(Part I) Response to Reviewer nW88

We sincerely thank you for the valuable feedback. We provide point-by-point responses below, with corresponding revisions highlighted in blue. We hope our responses adequately address your concerns and welcome any further comments or suggestions.

W1 & Q2: Task Significance.

We respectfully disagree that hierarchical visual recognition is an artificially constructed or redundant task. Hierarchical classification is a longstanding research area [1–4], aiming to recognize visual concepts at multiple semantic granularities while maintaining taxonomic consistency. Although leaf-level predictions can be mapped to their ancestors through deterministic lookup, this does not fully address the objectives of hierarchical visual recognition, for three reasons.

  • Hierarchical classification emphasizes multi-granularity recognition, enabling more flexible predictions. Different semantic levels may rely on different visual cues, allowing accurate coarse-level recognition even when fine-grained categories cannot be reliably distinguished [5]. In contrast, leaf-first lookup makes all ancestor predictions dependent on a single leaf-level decision, potentially propagating fine-grained errors to coarser levels. Multi-granularity recognition also accommodates different levels of desired specificity, as supported by the human preference study of Chang et al. (2021) [6].
    • To directly validate this distinction, we compare our method against a Leaf-only SFT + Lookup baseline on the matched subset where both methods misclassify Species, as shown in the table below.
    • Our method achieves 2.97–15.91 percentage-point higher ancestor accuracy across three evaluation splits, demonstrating its ability to preserve correct coarse-level predictions even when fine-grained recognition fails.
      | Dataset | Method | Genus | Family | Order | |---|---|---:|---:|---:| | Plantae Seen | Leaf-only SFT + Lookup | 42.84 | 63.66 | 68.80 | | | Ours | 52.24 | 79.57 | 84.00 | | Plantae Unseen | Leaf-only SFT + Lookup | 36.96 | 55.82 | 65.52 | | | Ours | 44.22 | 70.35 | 81.23 | | Animalia Unseen | Leaf-only SFT + Lookup | 24.97 | 60.99 | 86.83 | | | Ours | 36.03 | 67.24 | 89.80 |
  • Hierarchical classification considers error severity, enabling less severe and more meaningful predictions. Conventional FGVC primarily focuses on leaf-level accuracy, treating all incorrect predictions equally regardless of their semantic differences [7]. In contrast, hierarchical classification considers the relationships between categories, encouraging models to make less severe errors, such as confusing closely related species rather than species from entirely different families. This is particularly important in medical imaging, where different classification errors can have very different clinical consequences [8]. While taxonomy lookup can measure the severity of leaf-level errors, explicit hierarchical supervision can help models learn these relationships and reduce semantically distant mistakes, rather than merely assessing them after prediction.
  • In VLMs, hierarchical classification promotes the effective use of visual and taxonomic knowledge, enabling more structured visual understanding. Unlike conventional FGVC, hierarchical recognition with VLMs requires models to integrate visual evidence with the LLM's taxonomic knowledge across semantic levels. Tan et al. (2026) [9] reveal that VLMs struggle with hierarchical consistency even when their internal representations retain rich visual and taxonomic information, highlighting a gap between knowledge representation and effective prediction. Leaf-only classification with external taxonomy lookup bypasses this challenge rather than addressing the model's ability to produce visually grounded and hierarchically consistent predictions. Our framework directly targets this capability by eliminating historical textual shortcuts while explicitly preserving hierarchical relationships.
AuthorsOpenReview ↗

(Part II) Response to Reviewer nW88

W2 & Q1: Narrow Domain Scope and applicability beyond biology.

We respectfully point out that our method is not confined to biology but is applicable across domains with meaningful category hierarchies. Below, we explain why our framework can be applied across domains and present supporting experimental results on the non-biological Food-101 dataset.

  • Our method can be applied beyond biology because neither component relies on biological assumptions. Specifically, EICA controls access to preceding question–answer exchanges regardless of the domain, while the hierarchy loss constructs a compatibility matrix from explicitly specified parent–child relationships between adjacent-level candidates. Such hierarchies also exist beyond biology, including the manufacturer–family–variant hierarchy in FGVC-Aircraft [5,6], as well as ImgNet-Artifact and the hierarchical subset of Food-101 evaluated by Tan et al. (2026).
  • To examine applicability beyond biological taxonomies, we further evaluate our method on the hierarchical Food-101 benchmark, following the setting of Tan et al. (2026).
    • Setups. We use their published 84-category food hierarchy and retain the official Food-101 training and test partitions for these categories: 63,000 training images and 21,000 test images [9]. Using Qwen2.5-VL-3B, we compare the base model, HierSFT, TARA, and our method with identical candidate sets and independent queries at every level.
    • Results and analysis. As reported below, our method improves HCA over HierSFT by 3.99 percentage points and improves all three hierarchical overlap metrics, while slightly improving leaf-level accuracy. The substantial HCA gain narrows the gap between leaf-level accuracy and fully correct hierarchical predictions, demonstrating improved consistency across the complete hierarchical path. These results further validate the effectiveness of our framework beyond biological taxonomies. We have extended Sec. 5.2 to include the Food-101 results and analysis in Table 4.
      | Dataset | Hierarchy   | Method   | HCA | Acc$_{\mathrm{leaf}}$ | POR | S-POR | TOR | |:---:|:---:|---|---:|---:|---:|---:|---:| | | | Qwen2.5-VL-3B   | 43.05 | 87.85 | 73.86 | 67.82 | 52.81 | | Food-101 | 6-29-40-24   | HierSFT   | 84.95 | 92.11 | 93.58 | 92.28 | 89.01 | | | | TARA   | 71.95 | 91.55 | 88.18 | 85.59 | 79.25 | | | | Ours   | 88.94 | 92.27 | 95.07 | 94.65 | 92.32 |

W3: Over-emphasis on a Specific Artifact.

We respectfully point out that the textual shortcut studied in this work is not specific to TARA but represents a potential issue in serialized, teacher-forced VLM training for hierarchical recognition. For example, Tan et al. (2026) organize hierarchical supervision as multi-turn question–answer exchanges, while Li et al. (2026) generate coarse-to-fine taxonomic sequences within a single response during supervised fine-tuning. In this setting, subsequent predictions can access preceding questions and ground-truth answers alongside the input image. In hierarchical recognition, adjacent-level answers are linked by explicit taxonomic parent–child relationships, allowing a preceding parent-level answer to substantially constrain the candidate space for the current prediction. Consequently, the model may rely on preceding textual information without sufficiently examining visual evidence.

AuthorsOpenReview ↗

(Part III) Response to Reviewer nW88

W4 & Q3: Insufficient Baseline Comparisons & Missing Competitive Baselines.

Thank you for these suggestions. In the original manuscript, we did not include comparisons with FGVC methods because they address a different problem setting, as discussed in our response to Q2. To address your concern, we additionally compare our method with CLIP+SFT adapted to our hierarchical recognition setting.

  • Comparison with CLIP+SFT. To examine how the two recognition paradigms perform on the same hierarchical task, we compare our method with an adapted CLIP + SFT baseline under aligned training and evaluation protocols.
    • Setups. We apply hierarchical supervised fine-tuning to OpenAI CLIP ViT-L/14@336, using the same data and evaluation protocols as our main experiments. Each candidate category is encoded using the text prompt "a photo of a {category_name}." During evaluation, the model independently selects the candidate with the highest image–text similarity at each taxonomic level.
    • Results and analysis. As reported below, our method outperforms CLIP + SFT across all five metrics on all three evaluation splits. On Plantae Seen, our method improves HCA by 24.84 percentage points, compared with a 3.84-point gain in leaf-level accuracy. These results highlight the benefits of our approach in achieving fully correct hierarchical predictions beyond leaf-level classification.
      | Dataset | Method | HCA | Acc$_{\mathrm{leaf}}$ | POR | S-POR | TOR | |:---|---|---:|---:|---:|---:|---:| | | CLIP | 0.0027 | 30.84 | 26.71 | 23.28 | 11.72 | | Plantae Seen | CLIP + SFT | 3.45 | 44.89 | 56.25 | 47.99 | 38.80 | | | Ours | 28.29 | 48.73 | 79.48 | 72.33 | 69.08 | | | CLIP | 0.00 | 30.92 | 27.12 | 23.33 | 11.66 | | Plantae Unseen | CLIP + SFT | 1.84 | 32.94 | 48.66 | 40.97 | 30.57 | | | Ours | 23.05 | 46.86 | 74.73 | 66.88 | 62.58 | | | CLIP | 3.34 | 26.23 | 52.32 | 41.61 | 31.97 | | Animalia Unseen | CLIP + SFT | 2.66 | 26.84 | 49.47 | 40.14 | 29.99 | | | Ours | 22.96 | 47.72 | 75.79 | 68.52 | 64.58 |
  • Task and protocol differences from Fine-R1. We acknowledge Fine-R1 as a relevant recent FGVC method, but its task formulation and training–inference protocol differ from ours. Fine-R1 targets fine-grained category recognition, using CoT-based supervised fine-tuning followed by TAPO during training and explicit reasoning during inference. In contrast, our protocol independently queries each taxonomic level for a direct prediction, emphasizing not only leaf-level accuracy but also the correctness and consistency of complete hierarchical paths, evaluated using HCA and the three hierarchical overlap metrics (POR, S-POR, and TOR). Therefore, applying Fine-R1 under its original training and inference protocols would not provide a directly comparable evaluation of cross-level taxonomic consistency.

W5: Marginal General Task Improvements & Lack of Statistical Rigor.

  • Thank you for this valuable suggestion. To further demonstrate the statistical significance of our general-task improvements on a stronger backbone, we evaluate Qwen3-VL-8B-Instruct on MMStar using three training seeds for both HierSFT and our method under the same evaluation protocol.
  • As shown in the table below, results for HierSFT and our method are reported as means ± standard deviations. Our method improves accuracy over HierSFT by 1.14 percentage points, with a paired bootstrap 95% confidence interval of [0.13, 2.16]. We estimate this interval using 20,000 paired bootstrap replicates that jointly resample training seeds and test questions, with question resampling stratified by MMStar category. We have added these results and discussions in Table 12 and Appendix G.4.
    | Method | MMStar accuracy (%) | |---|---:| | Qwen3-VL-8B | 47.47 | | HierSFT | 66.02 ± 0.10 | | Ours | 67.16 ± 0.04 |

References
[1] A review of hierarchical classification. Journal of the Royal Statistical Society: Series A (General), 1987.
[2] A survey of hierarchical classification across different application domains. Data Mining and Knowledge Discovery, 2011.
[3] Semantic hierarchies for image annotation: A survey. Pattern Recognition, 2012.
[4] Review on hierarchical learning methods for large-scale classification task. SCIENTIA SINICA Informationis, 2018.
[5] Visually Consistent Hierarchical Image Classification. In ICLR, 2025.
[6] Your “Flamingo” is My “Bird”: Fine-Grained, or Not. In CVPR, 2021.
[7] Making Better Mistakes: Leveraging Class Hierarchies with Deep Networks. In CVPR, 2020.
[8] Hierarchical skin lesion image classification with prototypical decision tree. npj Digital Medicine, 2025.
[9] The LLM Bottleneck: Why Open-Source Vision LLMs Struggle with Hierarchical Visual Recognition. In CVPR, 2026.

Reviewer_Kg6xOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper investigates hierarchical visual recognition using VLMs, with a goal of a hierarchically consistent prediction through multiple levels of the hierarchy. The paper discusses that existing models encourage textual shortcuts. Therefore, the paper proposes a decoupled hierarchical training framework. The goal of decoupling is to separate hierarchical supervision from historical textual dependencies. The paper provides experiments on the iNat21-Plant and iNat21-Animal datasets, outperforming existing methods.

Critical strengths

The provided method performs well and outperforms existing methods in different tasks and metrics. Also, the proposed method is simple and effective. If I understand correctly, EICA only changes the attention mask, adds no parameters, and costs about the same as HierSFT (based on Table 6).

The paper points to a valid research question: how much of the hierarchical consistency comes from textual shortcuts and information provided in the text. The paper also provides studies to verify this point.

The paper has clear writing, motivating well and explaining well. It was easy to follow and understand.

Critical weaknesses

I have some main concerns about the analysis and experiments. Please check the questions for details.

Questions for authors

Finding 2 (Sec. 3.2, Fig. 2): I agree with the point raised by the paper. However, isn't Finding 2's result what one would expect? The parent answer makes the set of possible labels at the next level smaller. If you replace it with a wrong answer, the model is pushed to the wrong subset, so a big drop is not a surprise to me. How does this show a textual shortcut, and not just the effect of giving the model wrong information? It would help to also report the accuracy with no history (only the image and the question) and with the history but without the image. I am also confused by Table 8. Could the authors also add normal baseline numbers to the table? And why are the arrow directions for image swap and text swap different?

Table 5: What is the loss for each row? Is any of the λs set to 0 in the ablations, and does HierSFT use L_fmt?

Hyperparameters (Fig. 6): Figure 6 only studies λ_hier. How did you choose λ_hier (3.0 in the main experiments and 2.0 in the one-shot ones) and λ_fmt? Is it possible to show a similar analysis for the other hyperparameters? Also, would the authors explain the legend Acc leaf with two different-sized circles in Figure 6? What do the numbers and circles correspond to?

Leaf accuracy (Table 5): Adding the hierarchy loss lowers leaf accuracy on all three splits (49.19 to 48.73, 46.87 to 46.86, 48.32 to 47.72). It would be beneficial to provide more information on the behaviour of the loss and the accuracy drop.

Comparison with VL_Taxon and HiR2: I was wondering why the paper does not provide comparisons of performance with these two methods. It would be beneficial to also add a more thorough discussion on the differences between the methods.

Datasets: I am a bit confused by the datasets in the experiments. The paper provides experiments on Plantae Seen, Plantae Unseen, and Animalia Unseen. While the existing literature provides experiments on iNat21-Animal and iNat21-Plant. Are these the same or a subset? The information is provided in the dataset but is not very clear. Also, I am not sure if the iNat dataset is sufficient to prove the performance of the method.

Qualitative analysis: The paper provides only one successful example to compare with the baseline, which is not very informative. In what cases has the proposed method helped in classifying better? Also, the paper requires a failure case analysis. Where does the model still struggle? It would be good to provide more examples.

Candidate set choice: The paper discusses that the candidate set is chosen based on the highest similarity. I was wondering how the performance would be if the set were chosen randomly.

Writing note: It would be beneficial if the paper provides a clear definition of the shortcut and the candidate set in early parts of the paper.

5 comments
AuthorsOpenReview ↗

(Part I) Response to Reviewer Kg6x

We sincerely appreciate your valuable feedback. We provide point-by-point responses below. For your convenience, all revisions to the manuscript are highlighted in blue. Please kindly let us know if you have any further concerns.

Q1: Interpretation of Finding 2, additional control experiments, and clarification of Table 8.

Thank you for these constructive suggestions. We clarify the interpretation of Finding 2, provide additional control experiments to distinguish textual shortcuts from the effects of incorrect historical information, and revise Table 8 to clarify baseline accuracies and arrow directions.

  • Interpretation of Finding 2. Our analysis focuses on the extent to which historical text influences model predictions when it conflicts with visual evidence. Fig. 3 (Fig. 2 in the original submission) shows that, under the tested intervention settings, replacing historical answers causes a substantially larger accuracy drop than replacing the image, suggesting greater sensitivity to historical text.
  • Additional control experiments. Following your suggestion, we further examine this phenomenon without introducing incorrect information through two additional control experiments: No History and No Image.
    • No History retains the original image and the current question while removing all historical question–answer exchanges; No Image retains the correct historical question–answer exchanges and the current question while removing the image and its visual tokens.
    • The results are reported below. Across all three splits, removing historical exchanges (No History) causes a larger accuracy drop than removing visual input (No Image). The relatively high accuracy under No Image further indicates that historical textual answers provide substantial information for predicting subsequent taxonomic levels even without visual evidence, supporting the existence of a textual shortcut. We have revised Finding 2 in Sec. 3.2 and Appendix G.3 accordingly.
  • Clarification of Table 8. The original arrows indicate the desired directions of the accuracy drops: smaller drops under Text Swap (↓) are preferred because predictions should remain grounded in the unchanged image despite incorrect historical answers, while larger drops under Image Swap (↑) indicate greater sensitivity to visual input under the tested replacements. For clarity, we have revised Table 10 (originally Table 8) to include Correct Context accuracies and report intervention accuracies with their corresponding drops in parentheses. | Condition | Plantae Seen | Plantae Unseen | Animalia Unseen | |---|:---:|:---:|:---:| | Correct Context | 82.28% | 80.75% | 75.43% | | No History | 67.28% | 61.35% | 62.60% | | No Image | 75.28% | 74.10% | 64.43% |

Q2: Loss configurations in Table 5, coefficient settings in the ablations, and the use of the format loss in HierSFT.

Thank you for pointing out this ambiguity. We have revised Table 6 (Table 5 in the original submission) to explicitly report the loss coefficients and corresponding HCA results (%) for each method, as summarized below. Qwen2.5-VL-3B is evaluated directly to establish baseline performance, with no task-specific training loss. HierSFT and both masking ablations use the answer-token cross-entropy loss and the assistant end-of-turn format loss, with λans=1\lambda_{\mathrm{ans}}=1, λfmt=0.05\lambda_{\mathrm{fmt}}=0.05, and λhier=0\lambda_{\mathrm{hier}}=0. The final row additionally includes the hierarchy loss with λhier=3.0\lambda_{\mathrm{hier}}=3.0. HierSFT therefore also uses Lfmt\mathcal{L}_{\mathrm{fmt}}, with the same coefficient as our method.
| Method | λans\lambda_{\mathrm{ans}} | λfmt\lambda_{\mathrm{fmt}} | λhier\lambda_{\mathrm{hier}} | Plantae Seen | Plantae Unseen | Animalia Unseen | |:---|:---:|:---:|:---:|:---:|:---:|:---:| | Qwen2.5-VL-3B | — | — | — | 10.51 | 10.18 | 15.93 | | HierSFT | 1 | 0.05 | 0 | 23.56 | 19.09 | 21.04 | | EICA w/ Answer Masking | 1 | 0.05 | 0 | 23.39 | 18.73 | 20.92 | | EICA w/ Exchange Masking | 1 | 0.05 | 0 | 26.65 | 21.32 | 22.05 | | EICA + Hierarchy Loss | 1 | 0.05 | 3.0 | 28.29 | 23.05 | 22.96 |

AuthorsOpenReview ↗

(Part II) Response to Reviewer Kg6x

Q3: Hyperparameter selection, additional sensitivity analysis, and clarification of the legend in Fig.6.

  • Hierarchy-loss weight selection. We select λhier\lambda_{\rm{hier}} through grid search on a fixed validation set with three images per training species. Model selection is based on the trade-off between hierarchical consistency (HCA) and fine-grained recognition performance (Acc$_{\rm{leaf}}$). Guided by this trade-off, we use λhier=3\lambda_{\rm{hier}}=3 in the main experiments for improved HCA and λhier=2\lambda_{\rm{hier}}=2 in the one-shot setting.
  • Format-loss weight and additional ablation. Following [1], we supervise assistant end-of-turn markers to encourage termination after the answer. We assign this supervision a separate auxiliary weight, fixed at λfmt=0.05\lambda_{\rm{fmt}}=0.05 across HierSFT and all fine-tuned variants in Table 6. To examine its effect, we added a format-loss ablation in Appendix G.2 (Table 9). The results show that this supervision eliminates rare output-format errors, while recognition performance varies only slightly.
  • Circle sizes and numerical labels in Fig. 7 (Fig. 6 in the original submission). The x-coordinate of each point represents λhier\lambda_{\rm{hier}}, while the y-coordinate and the numerical label above each circle indicate HCA. The circle size and the percentage below each circle represent leaf-level accuracy, with larger circles indicating higher leaf-level accuracy. The two circles in the legend correspond to 48.51% and 49.19%, the minimum and maximum leaf-level accuracies in the figure, respectively, and serve as size references. The revised caption explicitly explains these visual encodings and their numerical references.

Q4: Adding the hierarchy loss lowers leaf accuracy on all three splits.

Thank you for this valuable comment. The hierarchy loss encourages taxonomically compatible predictions across levels. Most corrections address hierarchical prediction errors made by the model without the hierarchy loss, as reflected in the improvement in HCA. For example, as shown in Fig. 6 (Fig. 5 in the original submission), our method corrects an erroneous family prediction from Acanthaceae to Malvaceae while preserving the correct species prediction, Hibiscus syriacus. However, in a small number of cases, improving the overall hierarchical path may also change an originally correct leaf-level prediction, resulting in a slight trade-off in leaf accuracy. The increases in HCA and HCA_L indicate improved overall path correctness. In particular, higher HCA_L means that, among samples with correct leaf predictions, a larger proportion also have entirely correct ancestor paths.

Q5: Comparisons with VL-Taxon and HiR² and discussion of methodological differences.

Thank you for highlighting these methods. We cite both VL-Taxon and HiR² in Related Work (Sec. 2). We have not included direct comparisons with VL-Taxon and HiR² because we could not locate publicly released official implementations for reliable reproduction under our training and evaluation protocol. Below, we provide a more detailed discussion of the methodological differences and an additional reference comparison with published HiR² results.

  • VL-Taxon. VL-Taxon employs a two-stage pipeline trained with SFT and GRPO, using coarse-to-fine CoT reasoning to predict a leaf category and then using it as a prior for level-wise predictions, whereas our method enables independent prediction at each taxonomic level while addressing historical textual dependencies through EICA and encouraging cross-level consistency through a probabilistic hierarchy loss.
  • HiR². HiR² constructs hierarchy-specific visual representations and regularizes their geometry using taxonomic entailment and discriminative dispersive losses, whereas our framework isolates historical textual dependencies through EICA and encourages taxonomic consistency in the probability-distribution space.
  • Reference Comparison Following the HiR² Base/Novel Setting. We additionally evaluate our method on iNat21-Plant using Qwen2.5-VL-3B. Specifically, we partition the 4,271 species into 2,135 Base and 2,136 Novel species. HierSFT and our method are fully fine-tuned for one epoch using one image per Base species, and evaluated using one test image per species. Split1 denotes Base-to-Base evaluation, and Split2 denotes Base-to-Novel evaluation. Below, we report our results alongside the SFT and HiR² results published in [2]. These experimental details and results have been added in Appendix G.5. | Evaluation | Method | HCA | POR |S-POR|TOR| |---|---|---:|---:|---:|---:| |Split1|SFT|9.98|63.63|46.01|45.69| |Split1|SFT + HiR²|11.90|66.65|50.87|49.76| |Split1|HierSFT|16.39|70.78|60.77|56.21| |Split1|Ours|18.55|73.00|63.65|59.46| |Split2|SFT|10.86|63.74|46.15|45.16| |Split2|SFT + HiR²|13.06|67.16|50.69|49.69| |Split2|HierSFT|14.98|70.35|60.10|55.44| |Split2|Ours|17.93|72.97|63.84|59.49|
AuthorsOpenReview ↗

(Part III) Response to Reviewer Kg6x

Q6: Dataset correspondence, evaluation splits, and evaluation beyond iNaturalist.

  • Dataset correspondence and partitions. Thank you for pointing out this ambiguity. Our experiments use the same iNat21-Plant and iNat21-Animal benchmarks released by Tan et al. (2026). Specifically, Plantae Seen and Plantae Unseen are complementary subsets of iNat21-Plant, following their published species partition: Plantae Seen contains 37,710 evaluation images from the 3,771 species used for fine-tuning, while Plantae Unseen contains 5,000 images from 500 held-out species. Animalia Unseen is the complete released iNat21-Animal benchmark, containing 53,880 images from 5,388 species. We have clarified this in Sec. 5.1.
  • Evaluation beyond iNaturalist. To examine applicability beyond biological taxonomies, we further introduce the hierarchical Food-101 benchmark, following the setting of Tan et al. (2026).
    • Setups. We use their published 84-category food hierarchy and retain the official Food-101 training and test partitions for these categories: 63,000 training images and 21,000 test images [3]. Using Qwen2.5-VL-3B, we compare the base model, HierSFT, TARA, and our method with identical candidate sets and independent queries at every level.
    • Results and analysis. As reported below, our method improves HCA over HierSFT by 3.99 percentage points and improves all three hierarchical overlap metrics, while slightly improving leaf-level accuracy. The substantial HCA gain narrows the gap between leaf-level accuracy and fully correct hierarchical predictions, demonstrating improved consistency across the complete hierarchical path. These results further validate the effectiveness of our framework beyond biological taxonomies. We have extended Sec. 5.2 to include the Food-101 results and analysis in Table 4.
      |Dataset|Hierarchy|Method|HCA| Acc$_{\rm{leaf}}$|POR|S-POR|TOR| |:---:|:---:|---|---:|---:|---:|---:|---:| |||Qwen2.5-VL-3B|43.05|87.85|73.86|67.82|52.81| |Food-101|6-29-40-24|HierSFT|84.95|92.11|93.58|92.28|89.01| |||TARA|71.95|91.55|88.18|85.59|79.25| |||Ours|88.94|92.27|95.07|94.65|92.32|

Q7: Additional qualitative examples and failure case analysis.

We sincerely thank you for this helpful suggestion and have expanded the qualitative analysis in Appendix I to include additional success and failure cases.

  • The successful cases illustrate improvements in both fine-grained recognition and hierarchical consistency: our method corrects genus- and species-level predictions and rectifies erroneous ancestor predictions while retaining the correct species.
  • We also include a failure case in which both methods correctly predict the ancestor categories but confuse two species within the same genus, highlighting the difficulty of distinguishing closely related species when visual evidence is insufficient.

Q8: Performance with randomly selected candidate sets.

Thank you for this suggestion. We additionally evaluate HierSFT and our method using candidate sets that retain the ground-truth category and uniformly sample three distinct distractors from the same taxonomic level. Randomly sampled distractors are generally less visually or semantically similar to the target than similarity-selected ones, resulting in an easier recognition setting. As shown in the table below, our method consistently outperforms HierSFT on all five metrics across all three evaluation splits. These results demonstrate that the hierarchical recognition gains of our method persist under randomly selected candidate sets.

DatasetMethodHCAAcc$_{\rm{leaf}}$PORS-PORTOR
Plantae SeenHierSFT67.0290.3691.6284.7084.64
Ours69.3490.5092.7386.4586.42
Plantae UnseenHierSFT62.0690.1089.9381.9081.72
Ours64.3691.8090.7982.9782.87
Animalia UnseenHierSFT75.3091.9194.0689.8289.38
Ours77.4192.5294.6191.0890.51

Q9: Early definitions of textual shortcuts and candidate sets.

  • Thank you for this helpful suggestion. We have added an explicit definition of hierarchical textual shortcuts in Fig. 2 of Sec. 3.2, clarifying how preceding ground-truth question–answer exchanges can encourage reliance on taxonomic cues during training and reduce reliance on visual evidence.
  • We have also clarified the candidate set in Sec. 3.1: each level-specific question contains the ground-truth category and three distractor categories from the same taxonomic level, selected based on SigLIP image–text similarity.

References
[1] Right this way: Can VLMs Guide Us to See More to Answer Questions? In NeurIPS, 2024.
[2] Learning Taxonomic Trees with Hierarchical Representation Regularization for Large Multimodal Models. In ICML, 2026.
[3] The LLM Bottleneck: Why Open-Source Vision LLMs Struggle with Hierarchical Visual Recognition. In CVPR, 2026.

Comments on the paper

1 comment
Area_Chair_1UiDOpenReview ↗

Urgent Reminder: Reviewer Discussion Deadline

Dear Reviewers,

This is a friendly but urgent reminder that the discussion period will end on November 18, 2026.

Please carefully review the authors' rebuttals and actively participate in the ongoing discussions. If you have not yet responded to the authors or initiated a discussion, please do so as soon as possible.

In particular, reviewers are expected to address the authors' clarifications and responses, provide constructive follow-up feedback, and update their assessments and scores when appropriate.

Please ensure that your final evaluations reflect your independent judgment and take into account the information provided during the rebuttal and discussion process.

Active and timely participation is essential to ensuring a fair, thorough, and constructive review process.

Thank you for your time, effort, and cooperation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.