acceptodds
Under review as a conference paper at ICLR 2027

Learning Phylogeny-Constrained Relative Competition for Hierarchical Visual Recognition

Abstract

Hierarchical visual recognition (HVR) with large multimodal models (LMMs) aims to learn taxonomy-consistent representations. However, current target-centric post-training of LMMs only supervises the correct candidate, making inter-category discrimination hard to achieve. To bridge this gap, we propose PhyRC (Phylogeny-Constrained Relative Competition), which recasts the supervision from category identities to relative phylogenetic relations among competing candidates. Specifically, PhyRC introduces taxonomic ordinal constraints that encode discrete taxonomy as multi-level relative separation, requiring stronger separation for larger phylogenetic distances. These constraints further induce a phylogeny-consistent hierarchical metric space, where visual distances preserve taxonomic relatedness. Together, they enable the model to learn a transferable relational structure from phylogeny, allowing generalization to categories unseen during task-specific post-training. Extensive experiments across multiple LMM architectures, post-training objectives, and HVR benchmarks demonstrate consistent gains in hierarchical recognition and taxonomic consistency, including on task-unseen categories.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Hierarchical Visual Recognition, Large Multimodal Models, Phylogeny-Constrained Learning, Relational Supervision

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_jWMLOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes PhyRC (Phylogeny-Constrained Relative Competition), a new post-training method designed to improve taxonomic consistency and hierarchical visual recognition in Large Multimodal Models (LMMs). Although not novel (addressed in CLIP-like models), the central contribution is moving from category-based supervision to relationship-based supervision.

Critical strengths

The paper presents an extensive evaluation that goes beyond conventional accuracy metrics (i.e., hierarchcial consistency accuracy, leaf accuracy), assessing model robustness and taxonomic consistency under various training strategies and evaluation settings. This provides a more comprehensive understanding of model reliability and generalization, an this usually is not addresed in the emerging LMM benchmark paper courpus.

Critical weaknesses

The paper does not explicitly discuss its limitations. I encourage the authors to include a dedicated limitations section addressing the restricted taxonomic coverage, the focus on generative multimodal models, and the generalizability of the findings to other model architectures and more challenging biodiversity scenarios.

The dataset composition is not sufficiently clear, particularly regarding the number of images per species and the distribution of samples across taxonomic groups (~4.2K + 5.3k leaf categories, but over how many image samples? were they balanced). Given the well known class imbalance typically observed in biodiversity datasets, it would also be valuable to investigate whether the reported improvements in taxonomic consistency hold for underrepresented species, rather than being driven primarily by well-represented taxa.

The evaluation covers several open-weight multimodal models, including InternVL, Gemma, LLaVA, and Qwen. However, it would be valuable to extend the evaluation to proprietary models, such as GPT, Gemini, and Claude, to provide a broader comparison and assess whether the observed taxonomic consistency issues also occur in closed-source models.

Also, the table on page 8 is not corss-referenced.

Questions for authors

1.Why did the authors not consider CLIP-based models in their evaluation? Several strong biodiversity-specific vision-language models, such as BioCLIP and BioCLIP2, have demonstrated strong performance in taxonomic classification. Including these models would provide a valuable comparison and help assess whether the observed taxonomic consistency issues also extend to contrastive vision-language architectures.

2.Have the authors considered evaluating the models visually similar and difficult to distinguish species (i.e., Crypticbio[1])? Such an evaluation could provide further insights into the models' ability to maintain taxonomic consistency when fine-grained visual differences are limited.

3.Do models such as GPT, Gemini, and Claude exhibit similar taxonomic consistency problems?

[1] Manolache, Georgiana, Gerard Schouten, and Joaquin Vanschoren. "Crypticbio: A large multimodal dataset for visually confusing species." Advances in Neural Information Processing Systems 38 (2026).

7 comments
AuthorsOpenReview ↗

(1/4)Author rebuttal

Response to Critical Weaknesses: Dataset Composition, Evaluation Scope, and Limitations

We thank the reviewer for the constructive comments. Our dataset audit confirms balanced species-level sampling, while additional evaluations of API-based MLLMs provide further evidence that hierarchical recognition challenges extend beyond the model families evaluated in our main experiments. Below, we clarify the scope of these findings and outline the limitations we will discuss in a dedicated section of the revised manuscript.

(1) Dataset composition and class distribution.

To address the reviewer's concern that the dataset composition and the number of images per species were not sufficiently clarified in the original manuscript, we conducted an additional audit of the actual post-training and evaluation manifests and will clarify these details in the revision. Specifically, the iNat-Plant and iNat-Animal evaluation sets contain 4,271 and 5,388 images, respectively, with exactly one image per species. Their taxonomic distributions are summarized below.

DatasetImagesSpeciesGenusFamilyOrder
iNat-Plant4,2714,2711,70228685
iNat-Animal5,3885,3882,988715152
Total9,6599,6594,6901,001237

Importantly, the actual task-specific post-training dataset also contains one unique image per ground-truth species. The inat2021-mini dataset serves as the source pool, rather than representing the actual number of images used for post-training.

Consequently, conventional Head/Medium/Tail frequency-based analysis at the species level is not directly applicable to our actual training distribution. We therefore report macro-averaged accuracy and Task-Seen/Task-Unseen performance to provide complementary evaluations across species. Nevertheless, we acknowledge that the current balanced one-shot setting does not establish robustness under naturally long-tailed training distributions or account for variations in within-species visual diversity.

(2) Evaluation of multimodal large language models (MLLMs).

We acknowledge the reviewer's suggestion to extend the evaluation to proprietary multimodal models. Recent work by Tan et al. [1], The LLM Bottleneck: Why Open-Source Vision LLMs Struggle with Hierarchical Visual Recognition (CVPR 2026), has also reported taxonomic consistency difficulties in models such as GPT-4o.

Following this observation, we conducted additional evaluations using DeepSeek-V4.1-Flash and Kimi K2.6 through their official APIs on iNaturalist-2021. The results further confirm the presence of hierarchical recognition difficulties in multimodal large language models (MLLMs). Detailed protocols and results are provided in (4/4) Author Rebuttal.

(3) Limitations and Generalizability.

We will add a dedicated Limitations section addressing three aspects: (1) taxonomic coverage, currently focused on Plant and Animal; (2) architectural generalizability, as PhyRC primarily targets generative multimodal models; and (3) biodiversity scenarios, including naturally long-tailed distributions and within-species visual diversity. These directions will also inform future extensions of our work.

(4) Manuscript correction.

We thank the reviewer for pointing out this issue. We will add an explicit in-text cross-reference to the corresponding table on page 8 in the revised manuscript to improve clarity and readability.

[1] Tan, Yuwen, Yuan Qing, and Boqing Gong. "The llm bottleneck: Why open-source vision llms struggle with hierarchical visual recognition." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2026.

AuthorsOpenReview ↗

(2/4)Author rebuttal

Q1: Hierarchical Prediction Inconsistency in Contrastive Vision-Language Models

We thank the reviewer for this valuable suggestion. Our additional experiments reveal that hierarchical prediction inconsistency also exists in contrastive vision-language models. Despite achieving over 95% Species Accuracy, BioCLIP2 obtains substantially lower HCA (57.53% on Plant and 64.61% on Animal). BioCLIP exhibits a similar discrepancy. These findings suggest that the gap between fine-grained recognition and multi-level hierarchical recognition is not exclusive to generative MLLMs, but also exists in biodiversity-specific contrastive models.

Evaluation Protocol. We evaluate the official BioCLIP[1] and BioCLIP2[2] checkpoints using the same evaluation images, ordered four-candidate sets at each taxonomic level, ground-truth labels, and taxonomy mapping as in our main experiments. Following their native contrastive inference paradigm, we construct candidate descriptions using the prompt "a photo of a {category_name}." Image and text embeddings are independently L2-normalized, and candidates are ranked by cosine similarity at each taxonomic level. The resulting predictions are evaluated using the same hierarchical metrics as PhyRC.

Table. Evaluation of Contrastive Vision-Language Models on iNat21 (%).

ModelDatasetHCAPORS-PORTORLeaf
BioCLIPiNat-Plant21.9268.3662.7150.7889.30
BioCLIP2iNat-Plant57.5385.8882.2676.4395.41
BioCLIPiNat-Animal42.8478.8371.0961.7588.96
BioCLIP2iNat-Animal64.6188.7383.5178.1095.92

Analysis and Implications. The results reveal a substantial gap between Species Accuracy and hierarchical recognition performance. These results indicate that strong fine-grained species recognition does not necessarily translate into simultaneously correct predictions across taxonomic levels. Importantly, this phenomenon also occurs in biodiversity-specific contrastive vision-language models, suggesting that hierarchical prediction inconsistency is not specific to autoregressive generation architectures. We emphasize that this experiment investigates the generality of the observed phenomenon rather than comparing the effectiveness of different post-training methods. The results therefore extend our investigation of hierarchical recognition inconsistency across different vision-language architectures. We will incorporate these experiments and corresponding discussions into the revised manuscript.

[1] Stevens, Samuel, et al. "Bioclip: A vision foundation model for the tree of life." 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2024.

[2] Gu, Jianyang, et al. "Bioclip 2: Emergent properties from scaling hierarchical contrastive learning." Advances in Neural Information Processing Systems 38 (2025).

AuthorsOpenReview ↗

(3/4)Author rebuttal

Q2: Cross-Dataset Evaluation on Visually Confusing Species

We thank the reviewer for this valuable suggestion. To assess whether PhyRC maintains taxonomic consistency when fine-grained visual differences are limited, we conducted a zero-shot cross-dataset transfer evaluation on CrypticBio[1], a benchmark featuring visually similar and difficult-to-distinguish species. Without any CrypticBio-specific training or model selection, PhyRC outperforms RFT by 2.53 pp in Leaf Accuracy and 1.13 pp in HCA, demonstrating improved fine-grained recognition and full-path correctness under challenging visual conditions.

Experimental Setup. We conduct a zero-shot cross-dataset transfer evaluation using the official CrypticBio test records and released candidate groups. We compare three Qwen3-VL-2B-Instruct model variants: the original Vanilla model and the RFT and PhyRC variants, both post-trained exclusively on the same iNaturalist-2021 one-shot training split. RFT and PhyRC are evaluated over three matched seeds. No CrypticBio data are used for training, fine-tuning, checkpoint selection, or hyperparameter tuning. To ensure deterministic evaluation, we freeze a common ordering of the released candidate sets; for Fungi, which are not covered by the original plant/animal prompt, we replace only the query noun with “organism.” We apply the hierarchical metrics defined in the main paper using a shared, paper-faithful implementation. LEAF accuracy is computed on the fixed cohort of 3,613 readable images with valid leaf labels and candidate membership, whereas HCA, POR, S-POR, and TOR are computed on the shared subset of 2,187 readable images with complete six-rank taxonomic annotations. All model variants use identical sample cohorts, ordered candidate sets, per-sample prompts, preprocessing, scoring rules, and metric implementations.

Table. Zero-shot cross-dataset transfer results on CrypticBio (%).

MethodHCAPORS-PORTORLEAF
Vanilla Qwen3-VL-2B-Instruct24.9277.0473.6969.6224.72
RFT (3-seed mean ± std)25.19±0.8676.61±0.2573.85±0.1469.75±0.0925.46±0.35
PhyRC (3-seed mean ± std)26.32±1.3477.32±0.1474.64±0.1970.41±0.2027.99±0.52
Δ (PhyRC − RFT)+1.13+0.71+0.79+0.66+2.53

The results show that PhyRC consistently outperforms both Vanilla and compute-matched RFT in LEAF accuracy and HCA. In particular, the improvements in both leaf-level recognition and complete-path accuracy indicate that PhyRC provides benefits beyond species-level classification alone. PhyRC also achieves consistent gains over RFT across POR, S-POR, and TOR, although the magnitude of improvement varies across metrics. These findings provide evidence of improved cross-dataset generalization to visually confusing species and stronger taxonomic consistency under challenging fine-grained recognition conditions.

We will incorporate the CrypticBio evaluation, including the experimental protocol, comparative results, and corresponding analysis, into the revised manuscript. This additional evaluation further validates the generalization capability of PhyRC beyond the iNaturalist-2021 benchmark.

[1] Manolache, Georgiana, Gerard Schouten, and Joaquin Vanschoren. "Crypticbio: A large multimodal dataset for visually confusing species." Advances in Neural Information Processing Systems 38 (2025).

Reviewer_emiNOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper studies hierarchical visual recognition (HVR) with large multimodal models. The paper argues that current target-centric post-training of LMMs only supervises the correct candidate and proposes PhyRC. The method introduces two objectives: taxonomic ordinal constraints on candidate scores and phylogeny-consistent regularization of visual representations. Experiments on iNaturalist-2021 report consistent improvements across different backbones and training settings.

Critical strengths
  1. The paper is well-organized and easy to follow.
  2. The reported experimental results are promising, with consistent improvements across multiple LMMs and post-training objectives.
  3. The authors provide relatively comprehensive ablations and experimental analyses.
Critical weaknesses
  1. The main objective of this paper is not entirely clear. If the goal is to improve hierarchical prediction consistency, why not simply predict the leaf category and retrieve its ancestors from the taxonomy? This would guarantee a consistent prediction path by construction. Moreover, the proposed method does not support hierarchical early stopping prediction. For both seen and unseen categories, the model is required to select a category regardless of its confidence. In my view, a more meaningful capability for unseen categories would be to recognize when the model cannot reliably identify the leaf category and stop at a higher level where it is confident. If the goal is instead to improve fine-grained classification accuracy, the paper should compare against recent FGVR methods such as Fine-R1 (ICLR 2026) and DiVE-k (ICLR 2026).
  2. The proposed method shares important similarities with previous works such as TARA (CVPR 2026) and HiR² (ICML 2026). PhyRC combines a margin-based candidate-ranking loss with a taxonomy-aware metric-learning objective, both of which are well-established ideas.
Questions for authors
  1. The main experiments are conducted under a four-choice classification setting. This setting limits the practical value of the method. It is unclear whether the reported improvements would hold under open-ended recognition or a much larger candidate space.
  2. The use of LCA to model category relationships is rather heuristic. Phylogenetic relatedness does not necessarily correspond to visual similarity. At different taxonomic levels, incorrect candidates receive different penalties. What justifies this level-dependent weighting?
  3. The decision-level loss is applied directly to the final candidate scores. Compared with standard SFT, its main effect is to explicitly penalize incorrect candidate responses. I would like to see a direct comparison with a simple candidate-wise supervised objective that assigns a target of +1 to the correct candidate and −1 to each incorrect candidate, with equal weights and without LCA-based margins. This would help clarify whether the improvement comes from the proposed LCA modeling or simply from additional supervision on incorrect responses.
  4. According to Table 4, FCC, without taxonomy-dependent margins, already accounts for most of the improvement over the RFT baseline. The additional gain from taxonomy-calibrated separation (TCS) is relatively small, while PCVR also contributes only modest improvements in several settings.
  5. In my experience, improvements on unseen categories after task-specific fine-tuning can sometimes come from better adaptation to the question and answer format, rather than a genuine improvement in recognizing unseen categories. The current experiments do not sufficiently distinguish these possibilities. I suggest that the authors report the performance curves on both Task-Seen and Task-Unseen categories throughout training. If PhyRC is trained for N steps in the main experiments, what happens when training is extended to 2N steps? Please show the 2N-step performance curves on both Task-Seen and Task-Unseen categories. I believe such an analysis would help determine whether the method genuinely improves generalization to unseen categories or whether the reported gains are largely due to selecting a particularly favorable training checkpoint.
6 comments
AuthorsOpenReview ↗

(1/4)Author rebuttal

W1: Clarifying the Objective of HVR Beyond Leaf-level Recognition

We appreciate the reviewer’s important comments on the objective of hierarchical visual recognition, leaf-to-ancestor retrieval, hierarchical early stopping, and comparisons with FGVR methods. PhyRC aims to achieve visually grounded recognition at every taxonomic level by independently modeling multi-level predictions and explicitly learning cross-level taxonomic relationships. Unlike leaf-only prediction followed by ancestor retrieval, which guarantees structural path consistency but constrains parent predictions to the predicted leaf's lineage, PhyRC exploits visual evidence at each level, allowing correct parent-level predictions even when leaf identification fails. Our additional experiments confirm this advantage: on the identical subset where both methods misclassify the leaf, PhyRC improves parent-level accuracy by 7.18–17.10 pp across All and Task-Unseen splits. Confidence-aware hierarchical early stopping represents a complementary selective prediction capability beyond our fixed-depth setting, while FGVR methods primarily target leaf-level discrimination. We therefore evaluate PhyRC through both fine-grained accuracy and multi-level hierarchical recognition metrics.

Parent-level Accuracy on Shared Leaf-error Cases. To quantify this advantage, we compare parent-level accuracy on the identical subset where both Leaf-only + Retrieval and PhyRC make incorrect Leaf predictions. This matched-subset evaluation ensures a fair comparison by controlling for differences in leaf-level error cases.

Table. Parent-level accuracy on samples with incorrect Leaf predictions (%).

Evaluation SplitMethodGenusFamilyOrder
AllLeaf-only + Retrieval29.1857.3676.95
PhyRC40.9072.2384.13
Task-UnseenLeaf-only + Retrieval22.1052.6875.81
PhyRC38.3469.7885.00

PhyRC achieves higher parent-level accuracy at all three taxonomic levels, with particularly substantial gains on Task-Unseen species. These results demonstrate its ability to preserve useful higher-level predictions despite Leaf-level errors, mitigating error propagation from leaf-only ancestor retrieval.

Hierarchical early stopping. We would like to clarify that PhyRC focuses on fixed-depth hierarchical visual recognition, aiming to improve recognition accuracy across multiple taxonomic levels through explicit relational supervision. Confidence-based selection of prediction depth is outside the scope of this study.

Distinction from fine-grained visual recognition methods. Fine-R1 and DiVE-k primarily focus on fine-grained visual recognition (FGVR), emphasizing discrimination among fine-grained leaf categories. In contrast, our work addresses hierarchical visual recognition (HVR), which additionally requires independent predictions across multiple taxonomic levels and explicit modeling of inter-level relationships. Consequently, Leaf Accuracy alone cannot fully assess the capabilities targeted by PhyRC. A model may correctly identify a species while producing inconsistent predictions at the Genus, Family, or Order levels. Our evaluation therefore considers both fine-grained recognition and hierarchical metrics, including HCA, POR, S-POR, and TOR, to assess recognition accuracy and cross-level consistency. We acknowledge Fine-R1 and DiVE-k as relevant advances in FGVR and will further clarify their relationship to HVR in the revised manuscript.

Q1: Scalability to Larger Candidate Spaces and Scope of Open-Ended Recognition

We thank the reviewer for raising this concern. We clarify that PhyRC targets candidate-conditioned hierarchical recognition rather than open-ended recognition.

Regarding scalability, our original submission already includes a candidate-set scaling experiment in Appendix (Table 18), evaluating candidate sizes of K∈{2,4,8,16}K \in \{2,4,8,16\}. For (K=4K=4), we use the original training data, while other candidate configurations are constructed under matched training budgets. As reported in Table 18, PhyRC consistently outperforms RFT in HCA across all candidate sizes on both Plant and Animal, with improvements of 4.36–8.02 pp and 2.98–5.50 pp, respectively. These results demonstrate that the effectiveness of PhyRC extends beyond the original four-choice setting to candidate spaces of up to 16 categories (4× the original size).

We will further clarify the scope of PhyRC in the revision, explicitly distinguishing its candidate-set-based hierarchical recognition setting from open-ended recognition.

AuthorsOpenReview ↗

(2/4)Author rebuttal

Q2 & Q4: Rationale and Independent Contribution of Taxonomy-aware Supervision

We thank the reviewer for questioning the rationale of LCA-based margins and their incremental benefits beyond FCC. PhyRC uses LCA as a label-space prior, assigning larger margins to taxonomically distant candidates to reflect their greater hierarchical error costs, without assuming that taxonomic distance directly reflects visual similarity. To isolate the contribution of taxonomic relationships beyond FCC, we conduct two analyses: (1) randomized relation assignment, testing the importance of correct taxonomic assignments; and (2) taxonomic-distance-conditioned candidate analysis, examining how taxonomic relationships affect candidate competition. The results show that correct taxonomic assignments outperform randomized alternatives, while TCS increases target–negative margins and margin satisfaction across all taxonomic-distance groups. Together, these findings establish the contribution of phylogenetic supervision beyond generic candidate discrimination. PCVR further provides complementary visual representation regularization.

(1) Comparison with Randomized Relation-Assignment Controls

We compare TCS with two randomized relation assignment (RRA) controls that preserve candidate competition and margin values while permuting margins among negatives.

Table. FCC, TCS, and RRA results (%). Permutation seeds are 314159 (main paper, Table 4) and 42 (rebuttal run).

MethodDatasetHCA (%)Leaf (%)POR (%)S-POR (%)TOR (%)
FCCiNat-Plant16.8537.6572.7460.8358.90
TCS17.0838.2973.8359.8860.38
RRA (Seed42)16.6937.6073.2960.8559.82
RRA (Seed314159)16.9137.2372.9860.6559.58
FCCiNat-Animal14.1537.9871.0659.0959.33
TCS14.6039.2271.7360.9059.63
RRA (Seed42)14.3138.2171.6760.3259.51
RRA (Seed314159)12.8738.6371.1557.6158.80

Both RRA controls improve some metrics over FCC, but TCS outperforms them on HCA and Leaf Accuracy in both kingdoms and on nine of ten metric comparisons for each permutation seed. The additional RRA run confirms this pattern, supporting margin assignment based on actual LCA relationships. Table 4(a) further separates the contributions of TCS and PCVR. On Task-Unseen Animal, HCA drops from 12.01% (FCC) to 11.19% (TCS; -0.82 pp), then rises to 14.01% with Full PhyRC (+2.82 pp over TCS; +2.00 pp over FCC), illustrating their complementary contributions.

(2) Taxonomic-Distance-Conditioned Analysis of Candidate Competition

Using paired Seed-42 FCC/TCS scores on identical image–candidate pairs, we group negatives by taxonomic distance: Near (same genus), Medium (same family, different genera), Far (same order, different families), and Very Far (different orders). We evaluate pairwise ranking accuracy, target–negative score margins, and margin satisfaction.

Under FCC, Near candidates exhibit the lowest ranking accuracy in both kingdoms (53.50% on Plant and 54.03% on Animal), indicating greater discrimination difficulty among closely related species.

Table. Taxonomic-distance-conditioned effects of TCS over FCC.

DatasetDistanceScore Margin Δ (95% CI)Margin Satisfaction Δ (pp, 95% CI)
iNat-PlantNear+0.0430 [0.0308, 0.0551]+4.46 [3.04, 5.96]
Medium+0.0735 [0.0457, 0.1005]+1.98 [0.51, 3.47]
Far+0.1314 [0.0781, 0.1894]+2.09 [-0.60, 4.74]
Very Far+0.1155 [0.0913, 0.1389]+1.57 [0.34, 2.79]
iNat-AnimalNear+0.0434 [0.0250, 0.0622]+3.26 [1.34, 5.00]
Medium+0.1573 [0.1329, 0.1814]+1.69 [0.69, 2.75]
Far+0.1337 [0.1130, 0.1535]+1.80 [0.74, 2.88]
Very Far+0.0868 [0.0651, 0.1077]+1.88 [0.39, 3.26]

TCS increases target–negative margins in all eight dataset–distance groups (all 95% CIs exclude zero) and improves margin satisfaction in all eight (seven CIs exclude zero). Together, the RRA controls and distance-conditioned analysis show that LCA-based supervision calibrates candidate separation according to actual taxonomic relationships, beyond generic competition or arbitrary margin assignment.

AuthorsOpenReview ↗

(3/4)Author rebuttal

Q3: Comparison with Equally Weighted Candidate-wise Supervision

We thank the reviewer for suggesting a comparison with equally weighted candidate-wise supervision. To disentangle the contributions of candidate-wise supervision, LCA-based margins, and visual regularization, we progressively compare Candidate-wise ±1, FCC, TCS, and Full PhyRC under the same experimental protocol. PhyRC outperforms this baseline in both kingdoms, demonstrating the additional benefits of taxonomy-aware hierarchical supervision.

Candidate-wise ±1 Supervision. Given KK candidates, we set ty=+1t_y=+1 for the correct candidate and tj=−1t_j=-1 for each incorrect candidate (j≠yj\neq y).

We use the same length-normalized teacher-forced scoring function as PhyRC and apply a parameter-free transformation to each candidate score sjs_j.

s^j=2exp⁡(sj)−1.\hat{s}_j = 2\exp(s_j) - 1.

We optimize the equally weighted regression objective.

L±1=1K∑j=1K(s^j−tj)2.\mathcal{L}_{\pm 1} = \frac{1}{K}\sum_{j=1}^{K}(\hat{s}_j-t_j)^2.

This baseline assigns an identical weight of (1/K=0.25) to each candidate and does not use LCA, taxonomy-dependent margins, or visual regularization.

Experimental settings. We use seeds 42–44, the same training steps, and PhyRC's evaluation protocol.

Table. Comparison with Candidate-wise ±1 Supervision (mean ± std, %).

MethodiNat-Plant HCAiNat-Plant LeafiNat-Animal HCAiNat-Animal Leaf
Candidate-wise ±19.50±4.8722.63±11.507.93±4.0522.61±11.34
FCC16.79±0.4137.65±0.5113.33±0.8337.91±0.34
TCS17.08±0.1438.20±0.4814.28±0.5138.87±0.55
PhyRC17.78±0.3839.04±0.1614.84±0.2539.19±0.84

The progressive comparison reveals the distinct contributions of candidate supervision and taxonomy-aware objectives. Candidate-wise ±1 exhibits substantially larger standard deviations across the three seeds than the other methods, indicating less stable performance. FCC provides the largest initial improvements over Candidate-wise ±1 supervision, highlighting the effectiveness of listwise competition among candidate hypotheses. TCS further improves both Leaf Accuracy and HCA across the two kingdoms, supporting the additional contribution of LCA-based margin calibration. Incorporating PCVR into TCS yields further gains. These results clarify the complementary roles of full-candidate competition, taxonomy-conditioned separation, and visual representation regularization.

W2: Methodological Distinction from TARA and HiR²

We appreciate the reviewer pointing out the similarities to TARA and HiR². The key distinction lies in how taxonomic relationships are incorporated into learning: while TARA and HiR² primarily exploit taxonomy to align or organize visual representations, PhyRC explicitly uses LCA-derived relationships to determine candidate-specific decision margins through TCS, while jointly regularizing visual representations through PCVR. This enables PhyRC to connect taxonomy-conditioned candidate competition with phylogeny-aware visual representation learning.

Specifically, TARA aligns intermediate LMM visual representations with pretrained biological foundation model (BFM) features, and further aligns first answer-token representations with ground-truth labels to transfer hierarchical biological knowledge. HiR² constructs semantic-aware visual trees and employs hyperbolic taxonomic entailment and discriminative dispersive regularization to organize hierarchical visual representations. However, neither explicitly models candidate-specific separation requirements according to the taxonomic relationship between the ground-truth category and each competing negative candidate.

PhyRC addresses this gap through two complementary mechanisms. At the decision level, TCS uses LCA-derived relationships to assign candidate-dependent margins, explicitly calibrating the separation between the target and individual negative candidates. At the representation level, PCVR introduces local near–far ordinal constraints to regularize visual representations. This combination connects taxonomy-conditioned candidate competition with phylogeny-aware visual representation learning.

Our component-wise ablation results reported in the main paper (Table 4) show the complementary contributions of TCS and PCVR, while the UCS and RRA comparisons further demonstrate the importance of actual taxonomic relation assignments. We will further elaborate on the methodological distinctions between our approach, TARA, and HiR² in the revision.

Reviewer_Lma4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes PhyRC (Phylogeny-Constrained Relative Competition), a post-training framework that leverages hierarchical relationships among categories to improve fine-grained visual recognition in large multimodal models (LMMs).

The key idea is to move beyond simply supervising the correct answer and instead explicitly model how much higher the correct candidate should score than competing candidates, with the required separation determined by their phylogenetic distances. The framework combines taxonomy-conditioned ordinal constraints on candidate scores with hierarchical metric learning in the visual representation space.

Critical strengths
  1. Intuitive motivation and interpretable design. The paper extends the use of taxonomy from label-level supervision to modeling relative relationships among competing candidates. The motivation is intuitive, and the proposed approach is straightforward, interpretable, and effective without introducing substantial architectural complexity.

  2. Comprehensive experimental evaluation. The experiments cover multiple LMM backbones and include cross-dataset transfer, different candidate-set sizes, varying hierarchy depths, and robustness analyses across random seeds. These evaluations provide substantial empirical evidence for the effectiveness of the proposed approach.

  3. Compatibility with different post-training paradigms. PhyRC can be integrated with SFT, DFT, and RFT, demonstrating its flexibility and potential applicability across different post-training objectives.

Critical weaknesses
  1. Insufficient Comparisons with Hierarchy-Aware Representation Learning Methods. While comparisons with SFT, DFT, and RFT are appropriate for demonstrating the compatibility of PhyRC with different post-training paradigms, these methods are not specifically designed for hierarchical representation learning. Although TARA provides a relevant taxonomy-aware baseline, the experimental evaluation lacks broader comparisons with established hierarchy-aware approaches. In particular, the paper discusses several hyperbolic representation learning methods in the related work but does not include them in the experimental comparisons. Given that phylogeny-consistent hierarchical metric learning is presented as a key contribution, it remains unclear whether the proposed cosine-based near–far constraints offer advantages over existing approaches that explicitly model hierarchical structures. Suggestion: A controlled comparison using the same LMM backbone and training protocol, with alternative hierarchical representation objectives such as hyperbolic metric learning or hierarchical contrastive learning, would help clarify the effectiveness and novelty of the proposed representation-level supervision.

  2. Insufficient evidence for the independent contribution of taxonomy. The ablation results suggest that Full Candidate Competition (FCC) already accounts for a substantial portion of the overall performance improvements. The additional gains from taxonomy-calibrated separation are comparatively modest. The paper would benefit from a more convincing analysis of why explicitly incorporating phylogenetic relationships is necessary and how much they contribute beyond standard candidate-level discrimination.

  3. Potentially overstated claims about hierarchical metric learning. The proposed visual objective primarily enforces local near–far distance ordering through a triplet-style constraint. However, such local constraints do not necessarily guarantee that the learned representation space faithfully preserves the global hierarchical or tree-like structure of the taxonomy. Therefore, the characterization of the resulting representations as a "phylogeny-consistent hierarchical metric space" may be stronger than what the formulation and experiments establish.

  4. Limited scope of generalization. The Task-Unseen setting refers to categories excluded from task-specific post-training, rather than categories necessarily unseen during the foundation model's pretraining. Furthermore, the evaluation primarily focuses on multiple-choice recognition with explicitly provided candidate categories, rather than open-vocabulary recognition. These limitations make it difficult to assess how well the proposed method generalizes to more realistic open-ended recognition scenarios.

Questions for authors
  1. What is the essential benefit of incorporating phylogenetic relationships? Given that FCC already delivers substantial improvements, why is explicit phylogenetic supervision necessary? Could comparable improvements be achieved through conventional hard-negative mining, standard listwise cross-entropy, or adaptive margins based on semantic similarity? A more comprehensive comparison against these alternatives would help isolate the unique contribution of phylogeny-aware supervision.

  2. Why are hyperbolic embedding methods not considered as baselines? Since the paper explicitly aims to learn hierarchical visual representations that reflect taxonomic relationships, hyperbolic representation learning provides a natural and relevant alternative. Hyperbolic spaces are particularly suitable for modeling tree-like structures and hierarchical relationships. A comparison with representative hyperbolic embedding approaches would help clarify whether the proposed local metric constraints offer advantages over methods that explicitly model hierarchical geometry, and whether the claimed hierarchical representation properties are sufficiently supported.

4 comments
AuthorsOpenReview ↗

(1/3)Author rebuttal

W1 (Q2): Comparison with and Discussion of Hyperbolic Embedding Approaches

We appreciate the reviewer’s valuable suggestions on comparisons with established hierarchy-aware representation learning methods, particularly hyperbolic approaches. We have added controlled comparisons with representative hierarchy-aware methods. PhyRC achieves the highest Leaf Accuracy and HCA on both biological kingdoms. Geometric analyses further show improved global taxonomic-distance alignment and local hierarchical ordering. These results stem from PhyRC's core design: taxonomy-conditioned candidate competition and phylogeny-aware visual regularization. Specifically, TCS leverages LCA-derived relationships to adaptively calibrate separation margins between the ground-truth category and individual negative candidates, while PCVR imposes local near–far ordinal constraints on visual representations. By jointly optimizing candidate-level discrimination and visual relationships, PhyRC effectively exploits taxonomic information without requiring the visual embedding space to explicitly reconstruct the global taxonomy.

We compare four representative hierarchy-aware objectives from two families: hyperbolic geometry-based representation learning (PHE [1], HEC [2], and H-MERU [3]) and hierarchical contrastive learning (H-HCL [4]). To ensure a fair comparison, we adapted their respective representation-learning objectives to our Qwen3-VL-based candidate recognition framework, preserving their core learning principles while keeping the TCS decision-level supervision identical and replacing only the representation-level objective. All TCS-based variants use the same Qwen3-VL-2B-Instruct initialization, training data, candidate sets, and matched training budget.

Table. Comparison with hierarchy-aware representation learning methods (%). PHE, H-HCL, and H-MERU are represented in the Related Work, whereas HEC (Ganea et al., 2018) is newly introduced in this rebuttal. All TCS-based variants use identical decision-level supervision and differ only in their representation-level objectives.

MethodiNat-Plant LeafiNat-Plant HCAiNat-Animal LeafiNat-Animal HCA
RFT31.889.8829.359.47
TCS38.2917.0839.2214.60
TCS + PHE [1]38.0017.1638.1213.47
TCS + HEC [2]38.0217.0937.4513.66
TCS + H-MERU [3]38.1217.7537.7914.27
TCS + H-HCL [4]37.5116.3037.9913.88
TCS+PCVR (PhyRC)39.0817.9039.3614.97

Under the same experimental settings, PhyRC achieves the highest Leaf Accuracy and HCA on both iNat-Plant and iNat-Animal, outperforming representative hierarchy-aware methods based on Poincaré embeddings, hyperbolic entailment, and hierarchical contrastive learning. These results show that strong hierarchical recognition does not require explicitly modeling global taxonomic geometry in hyperbolic space. Instead, PhyRC effectively captures taxonomic relationships through local ordinal constraints in Euclidean space, providing a complementary approach that combines hierarchical discrimination with task-oriented visual representation learning.

Our additional geometric analyses further support the effectiveness of PhyRC. We select PHE as the representative hyperbolic baseline because it directly models hierarchical relationships through distances in the Poincaré ball. Using global correlation (higher indicates better alignment) and local violation rate (lower indicates better near–far ordering) as complementary metrics, PhyRC achieves a higher global correlation (0.2867 vs. 0.2834) and a lower local violation rate (0.0570 vs. 0.0663) than the Poincaré representation learned by TCS + PHE. These results suggest that taxonomy-guided local ordinal constraints capture both global taxonomic distance alignment and local hierarchical relationships. Combined with its higher Leaf Accuracy and HCA, PhyRC provides an effective alternative to explicitly constructing hyperbolic embeddings for hierarchical visual recognition.

We will incorporate these additional comparisons and the corresponding analysis into the revision.

[1] Nickel and Kiela. Poincaré Embeddings for Learning Hierarchical Representations. NeurIPS, 2017.

[2] Ganea et al. Hyperbolic Entailment Cones for Learning Hierarchical Embeddings. ICML, 2018.

[3] Desai et al. Hyperbolic Image-Text Representations. ICML, 2023.

[4] Kwon et al. Improving Visual Recognition with Hyperbolical Visual Hierarchy Mapping. CVPR, 2024.

AuthorsOpenReview ↗

(2/3)Author rebuttal

W2 (Q1): Evaluating the Independent Contribution of Phylogenetic Supervision

We thank the reviewer for this concern. We conducted two complementary experiments: (1) comparison with alternative candidate-level supervision strategies, showing that TCS improves Leaf Accuracy and HCA over FCC across both kingdoms (+0.29–0.96 pp), while HNM and SAM fail to match these gains across three seeds; and (2) taxonomic-distance-conditioned analysis, showing increased target–negative score margins and margin satisfaction across all eight dataset–distance combinations. Together, these results support the additional value of explicit phylogenetic supervision beyond generic candidate discrimination and semantic similarity.

(1) Comparison with Alternative Candidate-level Supervision Strategies

Using FCC's standard listwise cross-entropy as the baseline, we further evaluate HNM and SAM to examine whether hard-negative mining or semantic similarity-based adaptive margins can reproduce the benefits of phylogenetic supervision.

Hard-negative Mining (HNM). We select the highest-scoring incorrect candidate and optimize its relative score against the ground-truth candidate using a fixed-margin ranking objective.

LHNM=Softplus⁡(sj∗−sy+δ),δ=0.05.\mathcal{L}_{\mathrm{HNM}} = \operatorname{Softplus}\left(s_{j^\ast}-s_y+\delta\right), \quad \delta=0.05.

For each training instance, we select the highest-scoring incorrect candidate based on the current length-normalized candidate scores.

Semantic Adaptive Margin (SAM). We construct adaptive margins based on semantic dissimilarity between class embeddings, without using explicit taxonomy. Specifically, we extract class embeddings using the frozen text encoder of Qwen3-VL-2B-Instruct. Token-level representations are mean-pooled and L2-normalized to obtain class embeddings. We then compute cosine similarity between the ground-truth and negative candidate embeddings to derive normalized semantic dissimilarity (rjsemr_j^{\mathrm{sem}}). The adaptive margin is defined as follows.

mjsem=clip⁡(0.05+0.10rjsem,0.05,0.20).m_j^{\mathrm{sem}} = \operatorname{clip}\left(0.05+0.10r_j^{\mathrm{sem}}, 0.05, 0.20\right).

All methods are evaluated using the same evaluation protocol and hierarchical metrics as in the main paper. Results are reported as mean ± standard deviation over three seeds.

Table. Comparison of candidate-level supervision strategies and PhyRC (mean ±\pm std, %).

MethodLeaf (iNat-Plant)HCA (iNat-Plant)Leaf (iNat-Animal)HCA (iNat-Animal)
FCC37.65$\pm$0.5116.79$\pm$0.4137.91$\pm$0.3413.33$\pm$0.83
HNM32.34$\pm$0.6316.41$\pm$0.4634.45$\pm$0.7912.13$\pm$0.21
SAM33.22$\pm$0.5616.30$\pm$0.1234.78$\pm$0.9312.72$\pm$0.70
TCS38.20$\pm$0.4817.08$\pm$0.1438.87$\pm$0.5514.28$\pm$0.51
PhyRC39.04$\pm$0.1617.78$\pm$0.3839.19$\pm$0.8414.84$\pm$0.25

The results show that neither hard-negative mining nor semantic adaptive margins reproduce the improvements of TCS. In contrast, TCS consistently outperforms FCC across both kingdoms and both evaluation metrics, demonstrating the additional contribution of explicitly modeling phylogenetic relationships beyond standard listwise candidate discrimination.

(2) Taxonomic-Distance-Conditioned Analysis of Candidate Competition

We analyzed candidate competition across four taxonomic distances: Near (same genus), Medium (same family, different genus), Far (same order, different families), and Very Far (different orders). Under FCC, pairwise ranking accuracies were 53.50%, 62.12%, 69.69%, and 70.97% on Plant, and 54.03%, 69.41%, 70.98%, and 69.35% on Animal, respectively. Near candidates consistently exhibit the lowest accuracy in both kingdoms.

We further evaluated the changes in target–negative score margins and taxonomy-conditioned margin satisfaction under TCS relative to FCC. The results are reported below.

Table. Taxonomic-Distance-Conditioned Improvements of TCS over FCC.

AnalysisDatasetNearMediumFarVery Far
Score Margin ΔiNat-Plant+0.0430+0.0735+0.1314+0.1155
iNat-Animal+0.0434+0.1573+0.1337+0.0868
Margin Satisfaction Δ (pp)iNat-Plant+4.46+1.98+2.09+1.57
iNat-Animal+3.26+1.69+1.80+1.88

TCS increases both actual target–negative score margins and taxonomy-conditioned margin satisfaction across all eight dataset–distance combinations, demonstrating improved candidate discrimination and stronger alignment with the prescribed phylogenetic relationships.

AuthorsOpenReview ↗

(3/3)Author rebuttal

W3: Global Visual–Taxonomic Alignment through Local Phylogenetic Constraints

We appreciate the reviewer’s concern about whether local near–far constraints capture global hierarchical structure. Although PCVR operates through local ordinal constraints, visual representations inherently encode both species-specific features and shared characteristics of higher-level taxa. Interconnected local comparisons can encourage global taxonomic organization beyond individual triplets. We test this via (1) global and local taxonomic–visual alignment and (2) taxonomy-free visual tree reconstruction. The results demonstrate improved global distance alignment with PCVR and better taxonomy-free visual tree reconstruction with the complete PhyRC framework, providing empirical support for the local-to-global effect. We will clarify the rationale in the revision.

(1) Global and Local Taxonomic–Visual Structure Alignment

Using frozen checkpoints under a shared protocol, we measure Global Taxonomy–Visual Distance Correlation (Spearman correlation of taxonomic and visual distances) and Local Triplet Ordering Accuracy (proportion of triplets satisfying near–far constraints; Near: same genus, Far: different orders).

Table. Global and local representation evaluation.

EvaluationMethodiNat-PlantiNat-Animal
Global Taxonomy–Visual Distance CorrelationRFT0.6360.844
FCC0.6560.864
TCS0.6530.855
PCVR0.6650.868
PhyRC0.6720.876
Local Triplet Ordering AccuracyRFT0.8890.964
FCC0.9080.976
TCS0.9130.976
PCVR0.9150.980
PhyRC0.9160.981

PCVR improves both global distance alignment and local triplet ordering over RFT across both kingdoms, while PhyRC achieves the highest scores on both metrics.

(2) Global Visual–Taxonomic Structure Alignment via Tree Reconstruction

We reconstruct taxonomy-free visual trees via cosine-distance average-linkage HAC and compare them with ground-truth taxonomy using Cophenetic Correlation (Pearson/Spearman) and hierarchical clustering agreement (ARI/NMI) at Order, Family, and Genus levels.

Table. Taxonomic Tree Reconstruction Results.

DatasetMethodPearsonSpearmanOrder ARIFamily ARIGenus ARIOrder NMIFamily NMIGenus NMI
iNat-PlantVanilla0.1500.1250.1120.1280.1540.3470.5510.851
RFT0.1510.1210.1070.1490.1500.3330.5440.849
PhyRC0.1740.1430.1210.1760.1600.3740.5720.859
iNat-AnimalVanilla0.5570.4180.5130.2560.0590.6840.7260.890
RFT0.5610.4220.5030.2430.0510.6750.7260.891
PhyRC0.6570.5160.8410.2900.0650.7860.7770.892

PhyRC achieves the highest agreement with ground-truth taxonomy across all metrics in both kingdoms. These results indicate that PhyRC learns visual representations whose global distances and clustering better reflect the underlying taxonomy, supporting improved global visual–taxonomic alignment.

W4: Scope of Task-Unseen Generalization and Evaluation Limitations

Since foundation-model pretraining corpora are not fully transparent, we cannot determine whether Task-Unseen species appeared in pretraining or claim pretraining-level novelty. Our Task-Unseen setting is defined at the post-training stage: all images belonging to Task-Unseen species are strictly excluded from post-training supervision at every taxonomic level (Order, Family, Genus, and Species). Thus, our evaluation measures generalization to species entirely withheld from post-training. Meanwhile, our multiple-choice evaluation with explicitly provided candidates does not establish open-vocabulary or open-ended generalization. We will clarify these limitations in the revised manuscript.

Thank you for the detailed rebuttal and additional experiments.

  • W4 has been addressed: the clarified scope of Task-Unseen and the acknowledged open-vocabulary limitation are reasonable.
  • W1 is only partially addressed: PhyRC’s margins over the hierarchy-aware baselines are small, come from single runs, and fall within seed variance. On iNat-Animal, all four baselines also underperform TCS alone, so I am not sure the adapted baselines were adequately tuned.
  • W2 and W3 remain: TCS adds only +0.29 to +0.96 pp over FCC, and FCC alone accounts for most of the gain in global alignment. I suggest toning down the “phylogeny-consistent hierarchical metric space” claim.

These concerns closely overlap with Reviewer emiN’s, and they are insufficiently resolved. In particular, a leaf-only + ancestor-retrieval baseline would, by construction, reach an HCA equal to its Leaf accuracy, which is higher than PhyRC’s HCA. The rebuttal does not report this comparison. I therefore maintain my original rating.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.