acceptodds
Withdrawn from ICLR 2027

GHiML: VLM-Guided Hierarchical Multi-Label Learning with Applications to Benthic Imagery

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Vision language models (VLMs) encode substantial expert knowledge within their weights, yet their direct application to specialised classification domains such as benthic imagery remains limited. Meanwhile, vision transformers trained on these tasks rely predominantly on low-level visual latents (colours, textures, and shapes) that lack the higher-level reasoning humans employ for identification. We propose Guided Hierarchical Multi-Label (GHiML) learning, a pipeline that extracts conceptual latent variables from a VLM conditioned on training data, generates sample-relevant captions grounded in those variables, and fuses the resulting textual embeddings with image embeddings via cross-attention and a per-node gated router. This approach provides semantic information complementary to low-level visual features, without requiring the VLM to be fine-tuned or a generative model to synthesise training images. We also introduce the inverted HML score (), a similarity score that evaluates hierarchical multi-label predictions through graph traversal distance and a cardinality penalty, providing an interpretable alternative to existing measures, which are insensitive to realistic degradations in sparse, deep hierarchies. Experiments on BenthicNet, FathomNet and iNaturalist show that the router improves macro over the vision-only baseline by up to 6.7 pp and by up to 21.1 pp on in-domain benthic data, with gains persisting under distribution shift. Ablations locate the contribution of the text itself: on the multi-label biotic hierarchies, removing the per-sample text signal degrades while leaving macro unchanged, consistent with text acting through prediction cardinality rather than node discrimination, whereas iNaturalist, whose annotations are single-label, loses 18.7% of macro . Our results support the hypothesis that VLMs contain exploitable specialised knowledge that, when appropriately elicited, can augment classification in domains where direct visual learning is insufficient.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Computer Vision, Hierarchical Multi-Label Classification, Benthic Imagery

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_d8BxOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes GHiML for hierarchical multi-label classification. It uses a VLM to identify ecological attributes and generate structured image descriptions, then combines frozen image and text features through three classification branches and a learned router. The paper also introduces a hierarchy-aware evaluation metric. Experiments cover three benchmarks and show improvements over an image-only baseline.

Critical strengths
  1. The application is well motivated. Ecological descriptions could provide useful context for recognizing rare or visually ambiguous categories in benthic imagery.

  2. The evaluation includes multiple datasets, in-domain and out-of-domain benchmarks. The authors also provide useful analyses of text complementarity and the proposed metric's behavior.

Critical weaknesses
  1. Replacing image-specific text with a constant embedding does not hurt either F1 or $d_{\mathrm{inv}}$ (Tab 3) on BN-Su. On FathomNet, training with noise instead of text also gives similar results (Table 10). These results leave unclear how much the method benefits from the content of the ecological descriptions.

  2. Generic captions are compared only through k-NN analysis.

  3. The ecological descriptions are not validated for accuracy or visual support. Attributes such as predation strategy may come from prior knowledge rather than visible evidence, and the impact of incorrect descriptions is unclear.

  4. Figures 2 and 3 are difficult to read and lack a complete example showing how the descriptions improve a prediction.

Questions for authors
  1. How does the method compare with generic captions and constant text when trained using the same architecture and procedure?

  2. Have the authors checked the accuracy of the generated ecological descriptions, particularly for properties that cannot be directly observed in a single image?

Reviewer_j1dfOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces a VLM-Guided Hierarchical Multi-Label (GHiML) learning approach for benthic or underwater seabed images. They propose using a VLM that discovers concepts from training images and generates descriptions for each image. They embed these text-based descriptions using Qwen3 and Image embeddings from DINO and learn a multi-label classifier combining the two representations. They introduce a router that combines the two representations better. The authors also introduce a dinvd_{\mathrm{inv}} score to evaluate hierarchically predicted labels that combines hierarchical proximity and penalizes the excess predictions. They show empirical results across 3 datasets - BenthicNet, Fathomnet and iNaturalist.

Critical strengths
  • The paper presents an interesting ecological application and a hard task of hierarchical multi label classification with incomplete annotations.
  • They show the model's efficacy beyond seabed or benthic images to other datasets like FathomNet and iNaturalist
  • The ablation studies isolating the text only and image only branches are helpful
Critical weaknesses
  • The contributions of the text branch are not fully separated from model capacity and training. Table 3 replaces text embeddings with mean embeddings only at inference, after the model has been trained on meaningful descriptions.

  • Missing baselines. Although the authors show ablation results for different components, they only have 1image-only baseline. The paper would benefit from more hierarchical classification baselines.

  • Although the authors include a C-HMCNN-based image-only baseline and component ablations. It is missing comparisons with other established hierarchical classification methods. Additional comparisons would strengthen the model outputs.

  • A baseline that trains a classification head directly on the VLM’s visual/text features would help establish the benefit of the concept-generation and captioning pipeline. A parameter-efficient or LoRA fine-tuning baseline could also provide a useful comparison.

  • It is still unclear how much the proposed concept guidance contributes to the final performance. The paper compares generic captions only at the embedding level, without training the full model using them. Comparing both types of descriptions under the same architecture and training setup would help establish the benefit of concept selection.

  • The paper provides insufficient details about the iNaturalist prompts and selected concepts. Since this domain differs from benthic imagery, the authors should clarify how the prompts were adapted, which concepts were selected, and how these concepts contribute to classification. It will also be helpful to show how and what ecological concepts help in species classificaion

  • For iNaturalist, the results do not fully establish whether text improves discrimination between closely related species or mainly helps identify broader taxonomic groups. Reporting species-level accuracy and performance at different taxonomic levels would help clarify where the improvements occur, supported with additional baselines.

  • In the introduction and Figure 1, the paper motivates the method as addressing missing reasoning in vision models. However, the experiments mainly support that text provides coarse contextual information and reduces excess predictions. These findings do not directly establish improved reasoning. The motivation should be narrowed to match the evidence or supported by targeted experiments.

Questions for authors
  • Table 13 shows that removing text causes a 1.2% relative decrease in hierarchical F1 and a 21.7% relative decrease in d_{\mathrm{inv}}. Although the appendix explains their different sensitivity to excess predictions, can the authors better justify the chosen penalty strength and show how sensitive the conclusions are to it, particularly when annotations may be incomplete?

  • The ablation results present different seeds and summary statistics as the main results, and it is unclear which splits they are evaluated on.

Reviewer_Jer2OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

GHiML elicits ecological variables from a VLM, filters them by KL/EM scoring, generates per-image captions, and fuses image, text, and cross-modal streams through C-HMCNN heads and a per-node gated router. It also introduces dinv, an HML metric combining hierarchy distance with a cardinality penalty. Evaluation covers BenthicNet, FathomNet, and iNaturalist with multi-seed runs and significance tests.

Critical strengths

1.The pipeline is practical and consistently improves over image-only and text-only baselines across datasets and seeds, with stronger empirical rigor than typical domain papers. 2.dinv is a substantive contribution: it explicitly handles hierarchy proximity and prediction cardinality, with theoretical analysis and useful comparisons to hF1 and AP.

Critical weaknesses

1.Methodological novelty is limited. The authors call it a use-inspired pipeline rather than a new HML method; the core is mostly VLM captioning, frozen embeddings, cross-attention, and a gated mixture. Strong HML-specific baselines and budget-matched comparisons are missing. 2.The evidence that the router exploits per-sample text content is not fully convincing. Replacing captions with the mean embedding changes the mean text gate weight only slightly, yet text ablation strongly affects BN-Bio dinv. The mechanism needs clearer characterization.

Questions for authors

1.Please define Router† exactly and explain why it has 66M trainable parameters versus 10M for Router. Are Router and Router† comparisons budget-matched? 2.Given the small gate-weight change under mean replacement, can you report per-sample gate entropy or the text-branch contribution to fused logits? 3.dinv fixes k_P = 0.5. How sensitive are the reported rankings to this choice?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.