Learning Phylogeny-Constrained Relative Competition for Hierarchical Visual Recognition

Hierarchical visual recognition (HVR) with large multimodal models (LMMs) aims to learn taxonomy-consistent representations. However, current target-centric post-training of LMMs only supervises the correct candidate, making inter-category discrimination hard to achieve. To bridge this gap, we propose PhyRC (Phylogeny-Constrained Relative Competition), which recasts the supervision from category identities to relative phylogenetic relations among competing candidates. Specifically, PhyRC introduces taxonomic ordinal constraints that encode discrete taxonomy as multi-level relative separation, requiring stronger separation for larger phylogenetic distances. These constraints further induce a phylogeny-consistent hierarchical metric space, where visual distances preserve taxonomic relatedness. Together, they enable the model to learn a transferable relational structure from phylogeny, allowing generalization to categories unseen during task-specific post-training. Extensive experiments across multiple LMM architectures, post-training objectives, and HVR benchmarks demonstrate consistent gains in hierarchical recognition and taxonomic consistency, including on task-unseen categories.