acceptodds
Withdrawn from ICLR 2027

From Visual Facts to Semantic Concepts: Structuring Frozen Vision–Language Representations with Evidence–Concept Decomposition

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Frozen vision–language models provide transferable visual features that encode both directly observable cues and higher-level semantic information. We organize one frozen representation into shared, evidence, and concept pathways while keeping the pretrained backbone fixed. Raw visual fact descriptions are mapped to typed evidence tags covering entities, attributes, actions, relations, compositions, and scenes, while abstract concept labels are normalized separately. The model uses specialized residual branches, a concept-conditioned evidence gate, prototype-based predictors, and evidence-aware fusion. On a Flickr30k-based fact–concept benchmark, the full model reaches competitive balanced mAP (19.68 versus 19.29 for a strong shared MLP) while providing separate pathways that can be removed, perturbed, swapped, and probed. A fact bottleneck retains strong fact performance but drops to 7.41 concept mAP. Source removal, gate perturbation, evidence-type analysis, and probing show distinct roles for shared, evidence, and concept information, with the clearest gate effect appearing in the evidence-to-concept route. The structured pathways retain competitive prediction while making evidence and concept use directly measurable.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Vision-Language Models, Frozen Representations, Structured Evidence, Fact-Concept Hierarchy, Compositional Reasoning, Interpretable Representation Learning, Multimodal Learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_PJj5OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes SED, a structured prediction interface built on frozen CLIP image representations. It separates shared, evidence, and concept pathways, uses concept-conditioned gating to modify the evidence representation, and combines these pathways for fact and concept prediction through text-prototype heads. The evaluation uses Flickr30k-derived paired annotations with 603 evidence labels and 2,487 concept labels. It compares SED with alternative heads using the same frozen features and examines source removal, gate perturbation, concept substitution, and representation probes. Predictive performance is reported as close to that of a shared MLP, and the intended contribution is an inspectable and intervenable representation structure.

Critical strengths

The shared, evidence, and concept pathways expose representations that can be probed and modified separately. The paper avoids equating architectural separation with complete semantic disentanglement or causal mediation. This makes the proposed interface potentially useful even when its aggregate predictive accuracy is similar to a conventional shared head. Holding the frozen visual features and label space fixed is a sensible way to compare downstream structures. Source removal, gate perturbation, concept substitution, and probes address different aspects of the architecture. The authors also acknowledge the small predictive gap to the shared MLP rather than treating that gap alone as decisive evidence.

Critical weaknesses
  1. The seed-level comparison also needs reconciliation. Table 1 reports concept F1 of 39.10±0.3039.10 \pm 0.30 for SED and 38.86±0.1938.86 \pm 0.19 for the shared MLP, using the mean and sample standard deviation over three seeds. Table 2 reports a seed-42 difference of −1.08-1.08 points. Ordinary seed variation therefore does not explain the reported discrepancy.

  2. Table 3 introduces another full-model result, whose relationship to Tables 1–2 is insufficiently specified.

  3. The four refinement losses in Eq. 16 are named without enough information, and the provenance and construction of the paired fact–concept annotations are unclear.

  4. The final dataset counts after filtering, split construction, and the data used for vocabulary filtering also need specification. Since the contribution depends on the supervised semantic categories and the refinement procedure, the omissions materially limit reproducibility and the interpretation of the diagnostic findings.

Questions for authors
  1. Are the point estimates in Table 2 original test-set differences or means of bootstrap estimates? How the reported seed-42 difference relates to the three-seed statistics.

  2. Could you provide the referenced appendices, full definitions of the refinement objectives, and the annotation-generation procedure? How were evidence and concept labels produced and checked, what are the final split sizes, and were frequency-based vocabulary decisions made using training data only?

  3. The fixed/shuffled-gate interventions modify a gate that depends on both image-side and concept-side inputs, while substituting zcz_c also affects direct fusion and prediction routes. What happens when the concept contribution to the gate is disabled while retaining the image-side gate, or when zcz_c is substituted only inside the gate while its direct routes remain fixed?

Reviewer_tys7OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes SED, a framework for reorganizing frozen CLIP representations into visual evidence and abstract concepts. The main idea is to separate evidence and concept labels and introduce shared, evidence-oriented, and concept-oriented representation pathways. The concept features then guide evidence selection through a gating mechanism, and the resulting features are combined for downstream prediction. Besides, the authors also introduce branch removal, concept swapping, gate perturbation, and representation probing to examine how the pathways contribute. Through this design, the paper aims to provide a structured interface for analyzing evidence and concept use rather than achieve complete semantic disentanglement.

Critical strengths
  1. Relatively useful problem setting. The studied problem could be of interest for those aiming to utilize CLIP
  2. Relevant comparisons and diagnostics. The authors did compare several prediction heads using the same frozen features and examine the proposed pathways through complementary interventions.
Critical weaknesses
  1. Inconsistent reported statistics. Some reported results do not reconcile with the stated evaluation procedure. For example, balanced mAP is defined as the average of fact and concept mAP, but SED’s values of 17.71 and 21.42 give approximately 19.57 rather than the reported 19.68. Similarly, Table 1 reports concept F1 of 39.10 ± 0.30 for SED and 38.86 ± 0.19 for the shared MLP across three seeds, whereas Table 2 reports a seed-specific difference of −1.08. The reported variability is too small to explain that difference simply by selecting one seed. I encourage the authors to regenerate these tables under consistent evaluation settings and provide the per-seed results, checkpoints, and thresholds.

  2. The interventions do not isolate the proposed gating mechanism. The concept representation influences both the evidence gate and the final prediction through direct paths. Consequently, swapping this representation changes several information sources at once, so the reported 96.19% concept-change rate does not specifically demonstrate effective concept-conditioned evidence selection. Gate-only perturbations are more informative, but their effect on the final FusionScore is small: it changes from 0.4240 to 0.4213 with fixed gates and 0.4208 with shuffled gates. Although the intermediate diagnostic changes more substantially, its practical significance remains unclear. I think the authors should swap only the concept input to the gate while keeping the direct prediction paths fixed.

Questions for authors

Please see weakness

Reviewer_ZchQOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper addresses the problem that the existing CLIP visual representation merges the whole evidence and detailed conceptual part in a single pathway to the downstream task, which will weaken the concept visual representations. Thus, this paper is motivated to propose a separate branch to process these concept extractions through the text-prototype classifiers and fuse them in a late module. Experimental results on the Flickr30k-derived dataset demonstrate the effectiveness of the proposed separate branch and achieve better performance than the conventional baseline.

Critical strengths
  1. The motivation is clear and easy to follow.
  2. The performance of the proposed method is better than the baseline, as shown in the paper.
  3. The ablation study proves the effectiveness of each proposed branch.
Critical weaknesses
  1. The overall writing of this paper seems to be AI-generated. The wording is not normal, and the writing logic is strange, which makes it hard for the reviewer to understand the meaning.
  2. The proposed method is not novel, and existing works have fully explored this area. [1] already introduces a residual predictor to recover information missed by an incomplete concept vocabulary. For other components like a shared stem, separately supervised residual branches, channel-wise gates, prototype classifiers, and feature fusion, these are already common methods in existing work.
  3. In Table 3, this paper shows that replacing the learned gate with fixed or shuffled gates barely changes the final FusionScore: 0.4240 for the full model, 0.4213 with a fixed gate, and 0.4208 with shuffled gates. Is there no salient improvement from the gate? So what is the benefit of the learned gate?
  4. SED values in Table 1 are approximately 19.57, rather than the reported 19.68. This affects the claimed improvement over the shared MLP.
  5. The improvement in Table 1 is limited. Table 1 reports improvements over the shared MLP of only 0.31 points in fact mAP, 0.23 in concept mAP, 0.52 in fact F1, and 0.24 in concept F1.

[1] Liu, Y., Zhang, T., & Gu, S. (2025, June). Hybrid concept bottleneck models. In 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (pp. 20179-20189). IEEE.

Questions for authors

Please answer the weakness, especially about the novelty and performance part.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.