acceptodds
Under review as a conference paper at ICLR 2027

SEAR: Seeing What Matters via Structure-Guided Language

Abstract

Fine-grained and specialized image classification often relies on subtle differences in visual structure and appearance, including shape, color, texture, and localized defect patterns. Image-level labels provide limited guidance on which cues matter and where they occur. Pretrained vision-language models can make visual attributes explicit in text, but generic captions may not highlight the structural properties that distinguish classes. We introduce SEAR (Semantic Enhancement via Alignment and Reconstruction Network) to make discriminative visual structure explicit in language and fuse the resulting descriptions with image features, helping identify subtle but decisive cues. A frozen VLM generates these descriptions under explicit instructions to focus on parts, shapes, and local patterns. Alignment encourages semantic consistency between the two views, while reconstruction helps preserve complementary visual evidence that the descriptions may omit. A finite-sample analysis characterizes a text-quality threshold: generated text helps when the recoded structural signal outweighs generator noise and the additional estimation cost. Across eight benchmarks, SEAR improves mean F1-macro over the ResNet-101 image-only baseline by 0.32-7.68 percentage points, including 2.43 points on Steel Defect, where text alone performs poorly. On a 20-class CUB subset, SEAR with task-prompted descriptions also outperforms its counterpart using human-authored captions, suggesting that faithfully describing image content may not emphasize the attributes most useful for classification.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
vision-language models, language-assisted image classification, multimodal fusion, VLM-generated descriptions, contrastive alignment, generalization analysis

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_1WaWOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SEAR, an image classification framework that augments frozen visual representations with image-specific descriptions generated by a frozen vision–language model. The descriptions are produced using task-specific structural instructions and are used during both training and inference. Lightweight downstream modules are trained with classification, symmetric image–text contrastive alignment, and within-branch feature reconstruction objectives. The paper also presents a representation-bias decomposition and conditional comparisons of estimation rates. Experiments across multiple classification datasets report improvements over the chosen image-only baseline, alongside analyses of the training objectives, generator scale, reconstruction targets, and description sources.

Critical strengths
  1. A useful empirical observation about the complementary role of generated descriptions. I find the Steel Defect results particularly interesting: the text-only branch performs substantially worse than the image-only baseline, yet the full SEAR model improves on image-only classification (Table 1). This suggests that descriptions with limited standalone predictive value can still be useful when combined with visual features. The framework provides a practical setting for studying this effect, using frozen pretrained components and relatively small trainable modules without requiring a task-specific description generator to be trained.

  2. Relevant ablations that help characterize the method’s effectiveness and limitations. The SEAR-BASE/A/R comparisons help assess the contributions of the auxiliary objectives, with the full model generally improving on the basic fusion variant. The reconstruction-target and generator-scale experiments provide further insight into the method’s behavior, including the loss of the Steel Defect improvement with a smaller generator (Table 2). The paper also reports settings in which simpler objectives or human-authored descriptions perform better. These observations help clarify which design choices are useful and where the benefits are less consistent.

Critical weaknesses
  1. The experiments do not sufficiently isolate the contribution of structural guidance. My main concern is the absence of a controlled comparison establishing what the proposed structural instructions add. The paper does not compare its descriptions with generic captions under matched generation and training conditions, as acknowledged in Appendix A. On Steel Defect, image–text demonstrations are also included in the prompt, and their effect is not separated from that of the structural instructions (Section 3.2). The loss ablations assess the downstream objectives, so they do not resolve whether the gains arise from structural guidance, demonstrations, or the pretrained generator’s existing recognition capabilities. The human-caption comparison likewise changes the description source, and its outcome depends on the description budget (Appendix D.3). Since structural guidance is central to the paper’s motivation, its incremental value needs more direct evidence. The conditional analysis in Section 4 also leaves open whether the proposed instructions produce the representation properties assumed by the theory.

  2. The supplementary baseline statistics require clarification. The averages and variability reported for the ViT Image and CLIP Image baselines on Cats vs Dogs in Table 6 appear difficult to reconcile with the test-set description in Table 4 and the reporting conventions in Section 5.1 and Appendix D.1. This concern assumes that each run is evaluated on the full stated test set and that the summaries are computed as described. On a small binary test set, the possible prediction outcomes constrain both the attainable per-run scores and their aggregate variability. The reported combinations appear inconsistent with these constraints under the stated protocol. A difference in evaluation, aggregation, or error-bar definition could explain the discrepancy, but this is not currently specified. The individual-run results and a clear account of their aggregation are needed to interpret these particular comparisons.

Questions for authors
  1. Could you compare generic captions with structural descriptions while keeping the generator, text encoder, description budget, downstream model, and tuning settings fixed? A comparison on Steel Defect and a dataset without demonstrations would directly address the main concern.
  2. How does SEAR perform on Steel Defect when the structural instructions are used without image–text demonstrations?
  3. What test-set size, number of runs, aggregation procedure, and error-bar definition were used for the Cats vs Dogs results in Table 6?
  4. Could you provide the individual-run results for the ViT Image and CLIP Image baselines and explain how the reported averages and variability were obtained?
  5. Are the results in Tables 1 and 6 evaluated using the same protocol? If not, what differs between the two sets of experiments?
  6. What measurable notion of text quality or generator noise underlies the claimed threshold, and how does it relate to Equation (12)?
  7. Does the independence statement in Section 4.1 also apply to image-generated descriptions? If so, how is it reconciled with the information-preservation requirement in Assumption 2?
3 comments
AuthorsOpenReview ↗

Response to Reviewer 1WaW - Weakness 2 and Questions 3-5 - Cats vs Dogs statistics and evaluation

Thank you for recognizing the value of generated descriptions and for your detailed suggestions on prompt controls, statistical reporting, and the theoretical interpretation. We address these concerns through controlled comparisons, clarified evaluation, and explicit statistical models.

Weakness 2

The supplementary baseline statistics require clarification. The averages and variability reported for the ViT Image and CLIP Image baselines on Cats vs Dogs in Table 6 appear difficult to reconcile with the test-set description in Table 4 and the reporting conventions in Section 5.1 and Appendix D.1. This concern assumes that each run is evaluated on the full stated test set and that the summaries are computed as described. On a small binary test set, the possible prediction outcomes constrain both the attainable per-run scores and their aggregate variability. The reported combinations appear inconsistent with these constraints under the stated protocol. A difference in evaluation, aggregation, or error-bar definition could explain the discrepancy, but this is not currently specified. The individual-run results and a clear account of their aggregation are needed to interpret these particular comparisons.

Question 3

What test-set size, number of runs, aggregation procedure, and error-bar definition were used for the Cats vs Dogs results in Table 6?

Question 4

Could you provide the individual-run results for the ViT Image and CLIP Image baselines and explain how the reported averages and variability were obtained?

Question 5

Are the results in Tables 1 and 6 evaluated using the same protocol? If not, what differs between the two sets of experiments?

Response

  1. Appendix C.6, Table 8 corresponds to Table 6 in the original submission. We have rechecked the two Cats vs Dogs entries and corrected them in the revised table. Each baseline run evaluates all 100 Cats vs Dogs test images, using the split in Appendix B, Table 5 and seeds 42, 43, and 44. The reported values are the arithmetic mean and the population standard deviation (variance denominator 3) over these three runs.

  2. In seed order 42/43/44, CLIP Image scores 99.00/100.00/99.00, giving 99.33 ± 0.47; ViT Image scores 99.00/100.00/98.00, giving 99.00 ± 0.82. All values are F1-macro percentages; the two summaries are those reported in the revised Table 8.

  3. The evaluation protocol is the same: the SEAR results in Table 1 and CLIP/ViT image-baseline results in Table 8 share the complete test split, F1-macro metric, and argmax prediction rule. Their training losses differ: SEAR uses cross-entropy for this single-label task, whereas the CLIP Image and ViT Image baselines use binary cross-entropy.

Table R3.3. Cats vs Dogs individual runs underlying the summaries in manuscript Appendix C.6, Table 8.

BaselineSeed 42Seed 43Seed 44Mean ± SD
CLIP Image99.00100.0099.0099.33 ± 0.47
ViT Image99.00100.0098.0099.00 ± 0.82

Note. F1-macro (%) on all 100 test images in each run. The mean is the arithmetic average over the three runs; SD is the population standard deviation with variance denominator 3. Table 8 reports the aggregate values.

AuthorsOpenReview ↗

Response to Reviewer 1WaW - Weakness 1 and Questions 1-2 - Structural guidance and image–text examples

Weakness 1

The experiments do not sufficiently isolate the contribution of structural guidance. My main concern is the absence of a controlled comparison establishing what the proposed structural instructions add. The paper does not compare its descriptions with generic captions under matched generation and training conditions, as acknowledged in Appendix A. On Steel Defect, image-text demonstrations are also included in the prompt, and their effect is not separated from that of the structural instructions (Section 3.2). The loss ablations assess the downstream objectives, so they do not resolve whether the gains arise from structural guidance, demonstrations, or the pretrained generator’s existing recognition capabilities. The human-caption comparison likewise changes the description source, and its outcome depends on the description budget (Appendix D.3). Since structural guidance is central to the paper’s motivation, its incremental value needs more direct evidence. The conditional analysis in Section 4 also leaves open whether the proposed instructions produce the representation properties assumed by the theory.

Question 1

Could you compare generic captions with structural descriptions while keeping the generator, text encoder, description budget, downstream model, and tuning settings fixed? A comparison on Steel Defect and a dataset without demonstrations would directly address the main concern.

Question 2

How does SEAR perform on Steel Defect when the structural instructions are used without image-text demonstrations?

Response

  1. Human task knowledge enters SEAR through prompts specifying which parts, shapes, and defect patterns to describe. The newly added comparison in Section 5.4, Table 4 crosses Generic/Structured instructions with zero or seven image–text examples. The generator, text encoder, splits, maximum generation budget, downstream architecture, and tuning procedure are shared. Each prompt is tuned independently under the same search budget and validation criterion; the selected hyperparameters can therefore differ across prompts (Appendix C.2). This design separates the effects of instructions and examples. The direct-classification baseline in Section 5.2, Table 1 further separates the generator's own recognition ability from the contribution of its descriptions: direct Qwen classification reaches only 29.18 on Steel Defect.

  2. With the structural instruction and no examples, SEAR reaches 87.50 ± 0.57 F1-macro (%) on Steel Defect, 2.76 points above the Generic instruction (84.74 ± 0.50); values are means and standard deviations over three training seeds. Adding the seven examples improves the Structured result and lowers the Generic result, so the examples are effective when they accompany structural guidance. Appendix C.2, Table 7 extends the instruction-only comparison to the other seven datasets: task-specific instructions raise the mean on four, lower it on two, and match it on one. These controls compare complete instruction designs and show that their predictive contribution depends on the task and the guidance provided.

  3. Regarding the representation properties assumed in Section 4: the prompt comparisons and the fusion ablations in Sections 5.2 and 5.4, Tables 1, 3, and 4 quantify the practical contribution of structural guidance and complementary views. Human task knowledge specifies which attributes the descriptions should make explicit; the models in Appendix I illustrate how access to such attributes can simplify downstream regression. Section 4.3 explains the potential learning-efficiency benefit under explicit regression-family conditions. The Hölder condition concerns the unknown population conditional mean over its entire domain, so it cannot be verified conclusively from finite observations. Appendices I.2–I.4 make the mechanism concrete through joint models realizing linear and smoother nonparametric fusion, with strict bias gains and matching minimax bounds under the stated assumptions.

Table R3.1. Steel Defect prompt comparison (manuscript Section 5.4, Table 4).

InstructionNo examples7 examples
Generic84.74 ± 0.5083.72 ± 1.03
Structured87.50 ± 0.5788.15 ± 0.57

Note. Test F1-macro (%), mean ± SD over 3 seeds. Both instructions share the same 7 examples.

Table R3.2. Instruction-only comparisons on the other seven datasets (manuscript Appendix C.2, Table 7).

DatasetGenericTask-specific
EuroSAT96.78 ± 0.0696.99 ± 0.34
Blood Cell99.27 ± 0.2599.07 ± 0.34
Bone Fracture90.78 ± 0.4592.08 ± 0.76
Pneumonia96.43 ± 0.0795.66 ± 0.23
Beans93.96 ± 1.4396.29 ± 0.83
Oxford Pets94.69 ± 0.8795.72 ± 0.62
Cats vs Dogs99.33 ± 0.9499.33 ± 0.47

Note. Test F1-macro (%), mean ± SD over 3 seeds; no examples.

AuthorsOpenReview ↗

Response to Reviewer 1WaW - Questions 6–7 - Generation noise, Equation (12), and complementary structural information under dependence

Question 6

What measurable notion of text quality or generator noise underlies the claimed threshold, and how does it relate to Equation (12)?

Question 7

Does the independence statement in Section 4.1 also apply to image-generated descriptions? If so, how is it reconciled with the information-preservation requirement in Assumption 2?

Response

  1. In the revision, the text-quality threshold of the original submission is stated as the risk balance in Equation (12), and generation noise is given an explicit definition. The newly added analysis in Appendix I.2.8 connects structural-extraction accuracy with prediction risk. The analysis defines a model-based measure of text quality: the extracted structural attributes T~\widetilde T equal the true attributes TT plus independent, zero-mean isotropic Gaussian generation error, whose variance σgen2=k−1E∣T~−T∣2\sigma_{\mathrm{gen}}^2=k^{-1}\mathbb E\\|\widetilde T-T\\|^2 is the mean squared extraction error per coordinate. It is distinct from σξ2\sigma_\xi^2, the variance of the structural variation present in the image but omitted by the frozen representation. The former measures extraction accuracy; the latter, the amount of structure that a description can add. The value of a generated description therefore depends on how accurately it expresses target-relevant structure that is not fully captured by the frozen image representation.

  2. Equation (I.L.34) gives the representation-bias gain in this model, Δgen=σξ4∣cstr∣2/(σξ2+σgen2)\Delta_{\mathrm{gen}}=\sigma_\xi^4\\|\boldsymbol c_{\mathrm{str}}\\|^2/(\sigma_\xi^2+\sigma_{\mathrm{gen}}^2), where cstr\boldsymbol c_{\mathrm{str}} denotes the structural regression coefficients. The gain is positive and decreases as the generation noise increases. In Equation (12), the risk difference becomes Δgen−rf(n)\Delta_{\mathrm{gen}}-r_f(n), where rf(n)r_f(n) is the difference between the fused and image-only estimation errors. The generated view improves prediction risk when the bias gain exceeds this difference. The condition thus depends jointly on generation noise, sample size, and the regression problem.

  3. With noisy extraction, the conditional mean in Equation (I.L.35) retains the unknown structural function hLh_{\mathrm L}, so the estimation problem can remain nonparametric, and its estimation error is assessed for that conditional-mean family. With ideal extraction, I.2.5–I.2.6 establish the parametric estimation rate, and I.2.7 gives sufficient sample-size conditions for overall risk improvement. In the experiments, generated-text quality is varied through generator capacity (Section 5.4, Table 3(c)): with the 3B generator, SEAR returns to the image-only range, in line with a smaller bias gain under larger generation noise.

  4. No. The independence statement does not apply to image-generated descriptions. The revised Section 4.1 explicitly models generated structure as dependent on the image factors through vs=qω,ρ(u,vt)v^s=q_{\omega,\rho}(u,v^t). Human task knowledge in the prompt guides pretrained visual–language knowledge to select and express attributes of the same image. The structural view is thus a knowledge-guided expression of image content.

  5. Assumption 2 concerns information retained conditional on the frozen image representation, not on the image itself. Attributes already present in the image can be only partly captured by that compressed representation yet accessible through a generated description. Their common source is therefore compatible with complementary predictive information: the generator and image encoder can express different task-relevant aspects of the same image. Section 4.2, Theorem 1 quantifies this contribution through conditional projection, and Appendix I constructs joint models in which observing the omitted, target-relevant structure strictly reduces representation bias.

Reviewer_BQduOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper proposes a VLM alignment algorithm for fine-grained image classification. Existing methods do not pay sufficient attention to structure-based visual clues, and the method address the problem by textual information alignment with image encoder. Evaluation is conducted on multiple datasets, especially in industrial scenes. The proposed method outperforms image-only classification baselines and class-name prompt and matching baselines. The author also proved that generated textual structure reduces representation bias.

Critical strengths
  1. It proposes alignment and reconstruction losses so that image features benefit from textual information while maintaining their own information.
  2. It provides a proof that generated textual unified with image structure reduces representation bias, which is seldom analysed in multi-modal fusion domain.
Critical weaknesses
  1. The analysis of representation bias is not tested in the experiments, which makes the chapter isolated. More experiments are needed to evaluate the proof.
  2. The paper does not compare with GIST and TextTeacher, which use a similar structure to SEAR.
  3. The alignment uses an InfoNCE contrastive loss, which depends on batch-sampling settings, but the paper does not provide the relevant information.
Questions for authors
  1. For the textual embedding e^i,Gs\hat{e}^{s}_{i,G}, what is the detailed implementation?
  2. Is their any experiment that can evaluate some proof in chapter 4?
  3. What are the results on GIST and TextTeacher, which use a similar structure to SEAR?
2 comments
AuthorsOpenReview ↗

Response to Reviewer BQdu - Weakness 1 and Question 2 - Connecting theory and experiments

Thank you for recognizing the complementary roles of alignment and reconstruction. We have strengthened the connection between analysis and evaluation, added the requested baselines, and clarified text encoding and contrastive training.

Weakness 1

The analysis of representation bias is not tested in the experiments, which makes the chapter isolated. More experiments are needed to evaluate the proof.

Question 2

Is their any experiment that can evaluate some proof in chapter 4?

Response

  1. The revision relates the analysis in Section 4 to three comparisons on Steel Defect. First, SEAR raises F1-macro from 85.72 (Image-only) to 88.15 (Table 1), although the generated-text branch of SEAR alone reaches only 33.25 (Table 3(a)). The text therefore contributes by explaining target variation left unresolved by the image representation, which is the quantity measured by Δ\Delta in Theorem 1, rather than through its standalone accuracy. Second, without examples, structural guidance raises F1-macro from 84.74 to 87.50 (Table 4), indicating that the gain depends on whether the descriptions cover target-relevant structure. Third, replacing the 7B generator with the 3B generator returns SEAR to the image-only range (85.45; Table 3(c)), in line with the noisy-extraction analysis in Appendix I.2.8, where the bias gain decreases as generation noise increases. These observations are consistent with the structural-coverage mechanism analyzed in Sections 4.2–4.3.

  2. In the squared-loss analysis, Section 4.2 defines representation bias through the population-optimal predictor for each representation, and Equation (12) expresses the finite-sample risk difference as the representation-bias reduction minus the difference in estimation errors. Hölder classes provide a flexible nonparametric framework for studying how smoothness and dimension affect estimation rates, without specifying a parametric regression form. [3] The classification comparisons above provide empirical evidence consistent with the roles of structural coverage and complementary information in prediction.

  3. Appendix C.8 restores the synthetic check of the trade-off in Equation (12). In a linear model with one label-aligned generated coordinate, image-only and fused predictors are fitted by OLS for each generator-noise level and sample size, and their test MSE difference is averaged over 200 training draws. The check examines the sign and monotonic trends implied by the condition that the bias gain exceeds the difference in estimation errors.

  4. The new joint models in Appendices I.2 and I.3 provide explicit bias gains and matching minimax bounds. In Example I.2, the image-only conditional mean contains an unknown mapping from the frozen representation to structural attributes, whereas observing those attributes makes the fused conditional mean linear. This construction explains how explicit structure can simplify downstream prediction under the stated model assumptions. The bias gain is Δ=σξ2∣cstr∣2>0\Delta=\sigma_\xi^2\\|\boldsymbol c_{\mathrm{str}}\\|^2>0, the product of the omitted structural variance and the squared norm of the structural coefficients, and the estimation rate improves from n−2β/(2β+d0)n^{-2\beta/(2\beta+d_0)} on the image side to the fused linear rate (d0+k)/n(d_0+k)/n. Example I.3 retains nonparametric regression and establishes faster fused estimation when β′d0>β(d0+k)\beta'd_0>\beta(d_0+k). I.2.7 and I.3.8 combine the positive bias gains with the estimation-error bounds to obtain sufficient sample-size conditions for lower overall risk. The two examples thus connect the representation-bias decomposition in Section 4.2 with the estimation analysis in Section 4.3 under explicit assumptions.

Table R2.1. Steel Defect comparisons related to the structural-coverage mechanism (manuscript Tables 1, 3(a), and 4).

ExperimentSettingF1-macro (%)
Main comparison (Table 1)Image-only85.72 ± 0.60
Main comparison (Table 1)SEAR88.15 ± 0.57
Branch probe (Table 3(a))SEAR generated-text branch33.25 ± 1.44
No-example prompts (Table 4)Generic84.74 ± 0.50
No-example prompts (Table 4)Structured87.50 ± 0.57
Generator capacity (Table 3(c))SEAR with 3B generator85.45 ± 1.03

Note. Test F1-macro (%), mean ± standard deviation over three training seeds. The branch probe uses the projected SEAR text representation. The prompt comparison uses no examples.

[3] Stone (1982). Optimal global rates of convergence for nonparametric regression. Chen & Christensen (2015). Optimal uniform convergence rates and asymptotic normality for series estimators under weak dependence and weak conditions.

AuthorsOpenReview ↗

Response to Reviewer BQdu - Weaknesses 2–3 and Questions 1 and 3 - GIST and TextTeacher, text embeddings, and InfoNCE sampling

Weakness 2

The paper does not compare with GIST and TextTeacher, which use a similar structure to SEAR.

Question 3

What are the results on GIST and TextTeacher, which use a similar structure to SEAR?

Weakness 3

The alignment uses an InfoNCE contrastive loss, which depends on batch-sampling settings, but the paper does not provide the relevant information.

Question 1

For the textual embedding , what is the detailed implementation?

  1. We added both methods to the Steel Defect comparison in Section 5.3, Table 2. Test F1-macro (%) is 78.29±0.29 for the GIST adaptation, 61.04±1.14 for TextTeacher, and 88.15±0.57 for SEAR (means and standard deviations over three training seeds). SEAR exceeds the two baselines by 9.86 and 27.11 percentage points, respectively.

  2. Section 5.3, Table 2 specifies the configurations: GIST uses CLIP ViT-B/32, while TextTeacher and SEAR use ResNet-101 image backbones. The three methods differ in how they use language: GIST and TextTeacher use it during training and predict from images alone, whereas SEAR retains image-specific generated descriptions at inference. The comparison thus evaluates the complete systems with their stated backbones and input requirements, and the results support the value of retaining the generated view at inference.

Table R2.2. GIST, TextTeacher, and SEAR on Steel Defect (manuscript Section 5.3, Table 2).

MethodImage backboneInference inputF1-macro (%)
GIST adaptationCLIP ViT-B/32Image78.29 ± 0.29
TextTeacherResNet-101Image61.04 ± 1.14
SEARResNet-101Image + generated text88.15 ± 0.57

Note. Test F1-macro (%), mean ± standard deviation over three training seeds. The backbone column lists the image encoder; SEAR also uses a frozen Qwen2.5-VL generator to provide its textual view.

  1. Text embedding. Section 3.1 defines the representations, and Appendix C.1 details their implementation. Each description generated offline by Qwen2.5-VL-7B-Instruct is tokenized, with a limit of 512 tokens (Appendix C.2, Table 6), and its input token embeddings are mean-pooled into a 3,584-dimensional vector. Trainable projection heads map these vectors and 2,048-dimensional ResNet-101 features to 256 dimensions each. The generator and both encoders remain frozen; the projection, concatenation–MLP fusion, and reconstruction modules are trained jointly.

  2. Batch sampling. Section 3.3 specifies that training examples are shuffled into mini-batches each epoch, and Appendix C.1 reports a batch size of 32. In a full batch of image–description pairs, each anchor has one matched positive and 31 within-batch negatives in each direction. Both projected embeddings are ℓ2\ell_2-normalized, and the image-to-text and text-to-image InfoNCE losses are averaged. The temperature is selected per dataset by validation-based Optuna tuning; Appendix F, Table 12 reports the selected values.

Reviewer_8FNROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SEAR, which uses a frozen VLM to generate structure-guided image descriptions and combines them with visual features. It uses cross-modal alignment and feature reconstruction to improve fine-grained image classification.

Critical strengths
  1. The paper evaluates SEAR on 8 datasets covering different domains. That is quite comprehensive. SEAR outperforms the ResNet-101 image-only baseline on 8 datasets. The authors also provide ablation studies on different loss functions and analyze the impact of VLM size.
  2. The Steel Defect experiment shows that even when the generated text performs poorly on its own, it can still provide useful complementary information for image classification. I think this is an interesting finding.
Critical weaknesses
  1. The method mainly combines prompts, contrastive learning, and feature reconstruction, which are all existing techniques. The overall approach feels more like an engineering combination of existing components. Also, the paper does not directly compare general image descriptions with structure-guided descriptions. So it is unclear whether the improvement really comes from the structure-guided descriptions or simply from adding text information.
  2. Theorem 1 mainly shows that adding useful information can reduce prediction error, which is a relatively straightforward result. Theorem 2 assumes that the regression function becomes smoother after feature fusion, but the paper does not clearly explain why SEAR should satisfy this assumption. Also, the independence assumption about text-specific factors in Section 4.1 seems to need further clarification, since the text is actually generated from the input image.
Questions for authors
  1. Could the authors add a comparison between general image descriptions and structure-guided descriptions to better show the contribution of the structure-guided prompt?
  2. Since the method already uses Qwen2.5-VL-7B to generate image descriptions, why not use this model directly for classification? I suggest adding this baseline, or even stronger VLMs, to better demonstrate the advantages of SEAR.
  3. Could the authors further explain the independence assumption about text-specific factors in Section 4.1, and whether the smoothness assumption in Theorem 2 actually holds for SEAR?
3 comments
AuthorsOpenReview ↗

Response to Reviewer 8FNR - Weakness 1 and Question 1 - Contribution of structural guidance

We thank the reviewer for the careful reading and constructive feedback, which have helped us improve the clarity and quality of our manuscript.

Weakness 1

The method mainly combines prompts, contrastive learning, and feature reconstruction, which are all existing techniques. The overall approach feels more like an engineering combination of existing components. Also, the paper does not directly compare general image descriptions with structure-guided descriptions. So it is unclear whether the improvement really comes from the structure-guided descriptions or simply from adding text information.

Question 1

Could the authors add a comparison between general image descriptions and structure-guided descriptions to better show the contribution of the structure-guided prompt?

Response

  1. Our central contribution is to translate human task understanding into structural guidance for image classification. When the visual cues that determine the label are subtle and hard to distinguish, human experience helps identify the informative structures. SEAR encodes this understanding in a structural prompt, consisting of an instruction (Appendix D, Table 10) and optional image–text examples, which guides a frozen VLM to describe the relevant parts, shapes, and local patterns. The fusion design in Section 3.3 follows from the nature of these descriptions: because generated text may be incomplete or inaccurate, alignment emphasizes the content supported by both views, self-reconstruction preserves the complementary evidence that alignment may suppress, and classification supervision learns how to use the two views jointly. Each component thus addresses a specific requirement of learning from generated descriptions. Table 1 supports this design: the alignment-only and reconstruction-only variants each fail to exceed the image-only baseline on three datasets, whereas the full objective exceeds it on all eight.

  2. We added the requested comparison between Generic and Structured instructions in Section 5.4, Table 4. On Steel Defect, structural guidance raises F1-macro from 84.74±0.50 to 87.50±0.57 without examples and from 83.72±1.03 to 88.15±0.57 with the same seven examples, gains of 2.76 and 4.43 percentage points (means and standard deviations over three training seeds). On this dataset, Generic descriptions remain below the image-only baseline (85.72±0.60), whereas structure-guided descriptions exceed it; the improvement is therefore attributable to structural guidance rather than to the mere addition of text. The two instructions share the generator, text encoder, data splits, generation budget, downstream architecture, and tuning procedure; each setting is tuned independently under the same search budget and validation criterion (Appendix C.2, Table 6).

  3. On the other seven datasets, which use instructions alone (Appendix C.2, Table 7), task-specific instructions raise the mean F1-macro on four datasets, lower it on two, and match the Generic mean on Cats vs Dogs. Together, these results show that the benefit of structural guidance is task-dependent and that image–text examples are most effective when combined with a structural instruction.

Table R1.1. Steel Defect prompt comparison (manuscript Section 5.4, Table 4).

InstructionNo examples7 examples
Generic84.74 ± 0.5083.72 ± 1.03
Structured87.50 ± 0.5788.15 ± 0.57

Note. Test F1-macro (%), mean ± standard deviation over three training seeds. Both instructions use the same seven image–text examples in the 7-example setting.

Table R1.2. Instruction-only comparisons on the other seven datasets (manuscript Appendix C.2, Table 7).

DatasetGenericTask-specific
EuroSAT96.78 ± 0.0696.99 ± 0.34
Blood Cell99.27 ± 0.2599.07 ± 0.34
Bone Fracture90.78 ± 0.4592.08 ± 0.76
Pneumonia96.43 ± 0.0795.66 ± 0.23
Beans93.96 ± 1.4396.29 ± 0.83
Oxford Pets94.69 ± 0.8795.72 ± 0.62
Cats vs Dogs99.33 ± 0.9499.33 ± 0.47

Note. Test F1-macro (%), mean ± standard deviation over three training seeds. All settings use instructions without examples.

AuthorsOpenReview ↗

Response to Reviewer 8FNR - Question 2 - Direct Qwen classification

Question 2

Since the method already uses Qwen2.5-VL-7B to generate image descriptions, why not use this model directly for classification? I suggest adding this baseline, or even stronger VLMs, to better demonstrate the advantages of SEAR.

Response

  1. We added the suggested Qwen2.5-VL-7B direct-classification baseline to Section 5.2, Table 1. As specified in Appendix C.3, the frozen model receives each image and the public class list and predicts labels with greedy decoding; a deterministic normalization rule maps its outputs to the class labels. Direct Qwen classification and SEAR are evaluated on the complete test splits using F1-macro.

  2. SEAR exceeds direct Qwen classification on seven of the eight datasets; on Steel Defect, direct classification reaches 29.18, compared with 88.15±0.57 for SEAR. The exception is Cats vs Dogs, where direct classification reaches 100.00, consistent with the generator's strong recognition of these two familiar categories. The comparison clarifies the role of the generator in SEAR: on specialized tasks, it is used to describe discriminative attributes, while the supervised predictor learns the mapping from the visual and textual representations to the task labels from training data.

Table R1.3. Direct Qwen classification and SEAR (selected columns of manuscript Section 5.2, Table 1).

DatasetQwen2.5-VL-7B directSEAR
Steel Defect29.1888.15 ± 0.57
EuroSAT29.7496.99 ± 0.34
Blood Cell6.9999.07 ± 0.34
Bone Fracture41.9992.08 ± 0.76
Pneumonia27.7295.66 ± 0.23
Beans62.1796.29 ± 0.83
Oxford Pets90.5495.72 ± 0.62
Cats vs Dogs100.0099.33 ± 0.47

Note. Test F1-macro (%). Qwen direct uses frozen inference with greedy decoding; SEAR reports mean ± standard deviation over three training seeds.

AuthorsOpenReview ↗

Response to Reviewer 8FNR - Weakness 2 and Question 3 - Theoretical mechanisms and generated-view dependence

Weakness 2

Theorem 1 mainly shows that adding useful information can reduce prediction error, which is a relatively straightforward result. Theorem 2 assumes that the regression function becomes smoother after feature fusion, but the paper does not clearly explain why SEAR should satisfy this assumption. Also, the independence assumption about text-specific factors in Section 4.1 seems to need further clarification, since the text is actually generated from the input image.

Question 3

Could the authors further explain the independence assumption about text-specific factors in Section 4.1, and whether the smoothness assumption in Theorem 2 actually holds for SEAR?

Response

  1. Theorem 1 quantifies the contribution of generated structure: the reduction in representation bias equals Δ=E[(f~fused−f~img)2]\Delta=\mathbb E[(\tilde f_{\mathrm{fused}}-\tilde f_{\mathrm{img}})^2], the target variation that the generated representation explains beyond the frozen image representation. Making this gain explicit allows it to be weighed against the estimation cost: Equation (12) expresses the finite-sample risk difference as Δ−rf(n)\Delta-r_f(n), where rf(n)r_f(n) is the difference in estimation errors between the fused and image-only predictors. Appendices I.2.7 and I.3.8 give sufficient sample-size conditions for an overall risk improvement in the two examples.

  2. The smoothness condition in Theorem 2 formalizes the effect of semantic recoding (Section 4.3). When relevant structural attributes are not fully captured by the image representation, learning their relationship with the target can be difficult; mathematically, the image-only regression function may vary sharply with the image representation. Making these attributes explicit in the fused view can simplify the relationship between the inputs and the target, yielding a smoother regression function that is easier to estimate. This intuition is consistent with the role of representations in making explanatory factors more accessible for prediction. [1] For example, in Appendix I.2, the fused regression is linear in the image and structural coordinates, whereas the image-only regression must also learn the unknown relationship between them.

  3. Hölder classes are widely used in nonparametric regression to describe nonlinear functions with different degrees of smoothness without imposing a specific parametric form, and they provide a common framework for comparing how smoothness and dimension affect estimation rates, making them suitable for examining the learning efficiency of the image and fused representations. [2] Because Hölder regularity is a property of the unknown population regression function over its whole domain, it cannot be verified conclusively from finite observations; we therefore state Theorem 2 under Assumption 3. The newly added examples in Appendices I.2–I.3 show that this assumption is attainable and make the proposed mechanism explicit. Example I.2 turns image-side nonparametric regression into fused linear regression; Example I.3 retains nonparametric regression and yields a faster fused minimax rate when β′d0>β(d0+k)\beta'd_0>\beta(d_0+k). Both examples have matching upper and lower bounds, and Appendix I.4 summarizes their correspondence to Theorems 1 and 2.

  4. The experiments are consistent with this mechanism. Figure 1 in Section 3.2 illustrates semantic recoding: visually different scratches are described with stable terms such as longitudinal, linear, and groove. In Section 5.2, Table 1, F1-macro on Steel Defect is 42.63 for Text-only, 85.72 for Image-only, 86.31 for SEAR-Base, and 88.15 for SEAR. The descriptions thus provide complementary information even though they are weak standalone predictors, and the fusion objectives help make use of it.

  5. The revised Section 4.1 defines the generated structure as vs=qω,ρ(u,vt)v^s=q_{\omega,\rho}(u,v^t), which depends explicitly on the image factors. Complementarity is defined relative to the frozen image representation rather than the image itself: generated attributes may express target-relevant structure that this representation does not fully capture. Image-generated text is therefore compatible with the structural-coverage condition in Assumption 2.

Table R1.4. Complementary descriptions and fusion on Steel Defect (selected results from manuscript Section 5.2, Table 1).

PredictorF1-macro (%)
Text-only42.63 ± 0.49
Image-only85.72 ± 0.60
SEAR-Base86.31 ± 0.62
SEAR88.15 ± 0.57

Note. Mean ± SD, 3 seeds. SEAR-Base: same fusion architecture; alignment/reconstruction loss weights = 0.

[1] Bengio et al. (2013). Representation learning: A review and new perspectives.

[2] Stone (1982). Optimal global rates of convergence for nonparametric regression. Chen & Christensen (2015). Optimal uniform convergence rates and asymptotic normality for series estimators under weak dependence and weak conditions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.