- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes SEAR, an image classification framework that augments frozen visual representations with image-specific descriptions generated by a frozen vision–language model. The descriptions are produced using task-specific structural instructions and are used during both training and inference. Lightweight downstream modules are trained with classification, symmetric image–text contrastive alignment, and within-branch feature reconstruction objectives. The paper also presents a representation-bias decomposition and conditional comparisons of estimation rates. Experiments across multiple classification datasets report improvements over the chosen image-only baseline, alongside analyses of the training objectives, generator scale, reconstruction targets, and description sources.
Critical strengths
-
A useful empirical observation about the complementary role of generated descriptions. I find the Steel Defect results particularly interesting: the text-only branch performs substantially worse than the image-only baseline, yet the full SEAR model improves on image-only classification (Table 1). This suggests that descriptions with limited standalone predictive value can still be useful when combined with visual features. The framework provides a practical setting for studying this effect, using frozen pretrained components and relatively small trainable modules without requiring a task-specific description generator to be trained.
-
Relevant ablations that help characterize the method’s effectiveness and limitations. The SEAR-BASE/A/R comparisons help assess the contributions of the auxiliary objectives, with the full model generally improving on the basic fusion variant. The reconstruction-target and generator-scale experiments provide further insight into the method’s behavior, including the loss of the Steel Defect improvement with a smaller generator (Table 2). The paper also reports settings in which simpler objectives or human-authored descriptions perform better. These observations help clarify which design choices are useful and where the benefits are less consistent.
Critical weaknesses
-
The experiments do not sufficiently isolate the contribution of structural guidance. My main concern is the absence of a controlled comparison establishing what the proposed structural instructions add. The paper does not compare its descriptions with generic captions under matched generation and training conditions, as acknowledged in Appendix A. On Steel Defect, image–text demonstrations are also included in the prompt, and their effect is not separated from that of the structural instructions (Section 3.2). The loss ablations assess the downstream objectives, so they do not resolve whether the gains arise from structural guidance, demonstrations, or the pretrained generator’s existing recognition capabilities. The human-caption comparison likewise changes the description source, and its outcome depends on the description budget (Appendix D.3). Since structural guidance is central to the paper’s motivation, its incremental value needs more direct evidence. The conditional analysis in Section 4 also leaves open whether the proposed instructions produce the representation properties assumed by the theory.
-
The supplementary baseline statistics require clarification. The averages and variability reported for the ViT Image and CLIP Image baselines on Cats vs Dogs in Table 6 appear difficult to reconcile with the test-set description in Table 4 and the reporting conventions in Section 5.1 and Appendix D.1. This concern assumes that each run is evaluated on the full stated test set and that the summaries are computed as described. On a small binary test set, the possible prediction outcomes constrain both the attainable per-run scores and their aggregate variability. The reported combinations appear inconsistent with these constraints under the stated protocol. A difference in evaluation, aggregation, or error-bar definition could explain the discrepancy, but this is not currently specified. The individual-run results and a clear account of their aggregation are needed to interpret these particular comparisons.
Questions for authors
- Could you compare generic captions with structural descriptions while keeping the generator, text encoder, description budget, downstream model, and tuning settings fixed? A comparison on Steel Defect and a dataset without demonstrations would directly address the main concern.
- How does SEAR perform on Steel Defect when the structural instructions are used without image–text demonstrations?
- What test-set size, number of runs, aggregation procedure, and error-bar definition were used for the Cats vs Dogs results in Table 6?
- Could you provide the individual-run results for the ViT Image and CLIP Image baselines and explain how the reported averages and variability were obtained?
- Are the results in Tables 1 and 6 evaluated using the same protocol? If not, what differs between the two sets of experiments?
- What measurable notion of text quality or generator noise underlies the claimed threshold, and how does it relate to Equation (12)?
- Does the independence statement in Section 4.1 also apply to image-generated descriptions? If so, how is it reconciled with the information-preservation requirement in Assumption 2?
3 comments
Response to Reviewer 1WaW - Weakness 1 and Questions 1-2 - Structural guidance and image–text examples
Weakness 1
The experiments do not sufficiently isolate the contribution of structural guidance. My main concern is the absence of a controlled comparison establishing what the proposed structural instructions add. The paper does not compare its descriptions with generic captions under matched generation and training conditions, as acknowledged in Appendix A. On Steel Defect, image-text demonstrations are also included in the prompt, and their effect is not separated from that of the structural instructions (Section 3.2). The loss ablations assess the downstream objectives, so they do not resolve whether the gains arise from structural guidance, demonstrations, or the pretrained generator’s existing recognition capabilities. The human-caption comparison likewise changes the description source, and its outcome depends on the description budget (Appendix D.3). Since structural guidance is central to the paper’s motivation, its incremental value needs more direct evidence. The conditional analysis in Section 4 also leaves open whether the proposed instructions produce the representation properties assumed by the theory.
Question 1
Could you compare generic captions with structural descriptions while keeping the generator, text encoder, description budget, downstream model, and tuning settings fixed? A comparison on Steel Defect and a dataset without demonstrations would directly address the main concern.
Question 2
How does SEAR perform on Steel Defect when the structural instructions are used without image-text demonstrations?
Response
-
Human task knowledge enters SEAR through prompts specifying which parts, shapes, and defect patterns to describe. The newly added comparison in Section 5.4, Table 4 crosses Generic/Structured instructions with zero or seven image–text examples. The generator, text encoder, splits, maximum generation budget, downstream architecture, and tuning procedure are shared. Each prompt is tuned independently under the same search budget and validation criterion; the selected hyperparameters can therefore differ across prompts (Appendix C.2). This design separates the effects of instructions and examples. The direct-classification baseline in Section 5.2, Table 1 further separates the generator's own recognition ability from the contribution of its descriptions: direct Qwen classification reaches only 29.18 on Steel Defect.
-
With the structural instruction and no examples, SEAR reaches 87.50 ± 0.57 F1-macro (%) on Steel Defect, 2.76 points above the Generic instruction (84.74 ± 0.50); values are means and standard deviations over three training seeds. Adding the seven examples improves the Structured result and lowers the Generic result, so the examples are effective when they accompany structural guidance. Appendix C.2, Table 7 extends the instruction-only comparison to the other seven datasets: task-specific instructions raise the mean on four, lower it on two, and match it on one. These controls compare complete instruction designs and show that their predictive contribution depends on the task and the guidance provided.
-
Regarding the representation properties assumed in Section 4: the prompt comparisons and the fusion ablations in Sections 5.2 and 5.4, Tables 1, 3, and 4 quantify the practical contribution of structural guidance and complementary views. Human task knowledge specifies which attributes the descriptions should make explicit; the models in Appendix I illustrate how access to such attributes can simplify downstream regression. Section 4.3 explains the potential learning-efficiency benefit under explicit regression-family conditions. The Hölder condition concerns the unknown population conditional mean over its entire domain, so it cannot be verified conclusively from finite observations. Appendices I.2–I.4 make the mechanism concrete through joint models realizing linear and smoother nonparametric fusion, with strict bias gains and matching minimax bounds under the stated assumptions.
Table R3.1. Steel Defect prompt comparison (manuscript Section 5.4, Table 4).
| Instruction | No examples | 7 examples |
|---|---|---|
| Generic | 84.74 ± 0.50 | 83.72 ± 1.03 |
| Structured | 87.50 ± 0.57 | 88.15 ± 0.57 |
Note. Test F1-macro (%), mean ± SD over 3 seeds. Both instructions share the same 7 examples.
Table R3.2. Instruction-only comparisons on the other seven datasets (manuscript Appendix C.2, Table 7).
| Dataset | Generic | Task-specific |
|---|---|---|
| EuroSAT | 96.78 ± 0.06 | 96.99 ± 0.34 |
| Blood Cell | 99.27 ± 0.25 | 99.07 ± 0.34 |
| Bone Fracture | 90.78 ± 0.45 | 92.08 ± 0.76 |
| Pneumonia | 96.43 ± 0.07 | 95.66 ± 0.23 |
| Beans | 93.96 ± 1.43 | 96.29 ± 0.83 |
| Oxford Pets | 94.69 ± 0.87 | 95.72 ± 0.62 |
| Cats vs Dogs | 99.33 ± 0.94 | 99.33 ± 0.47 |
Note. Test F1-macro (%), mean ± SD over 3 seeds; no examples.
Response to Reviewer 1WaW - Questions 6–7 - Generation noise, Equation (12), and complementary structural information under dependence
Question 6
What measurable notion of text quality or generator noise underlies the claimed threshold, and how does it relate to Equation (12)?
Question 7
Does the independence statement in Section 4.1 also apply to image-generated descriptions? If so, how is it reconciled with the information-preservation requirement in Assumption 2?
Response
-
In the revision, the text-quality threshold of the original submission is stated as the risk balance in Equation (12), and generation noise is given an explicit definition. The newly added analysis in Appendix I.2.8 connects structural-extraction accuracy with prediction risk. The analysis defines a model-based measure of text quality: the extracted structural attributes equal the true attributes plus independent, zero-mean isotropic Gaussian generation error, whose variance is the mean squared extraction error per coordinate. It is distinct from , the variance of the structural variation present in the image but omitted by the frozen representation. The former measures extraction accuracy; the latter, the amount of structure that a description can add. The value of a generated description therefore depends on how accurately it expresses target-relevant structure that is not fully captured by the frozen image representation.
-
Equation (I.L.34) gives the representation-bias gain in this model, , where denotes the structural regression coefficients. The gain is positive and decreases as the generation noise increases. In Equation (12), the risk difference becomes , where is the difference between the fused and image-only estimation errors. The generated view improves prediction risk when the bias gain exceeds this difference. The condition thus depends jointly on generation noise, sample size, and the regression problem.
-
With noisy extraction, the conditional mean in Equation (I.L.35) retains the unknown structural function , so the estimation problem can remain nonparametric, and its estimation error is assessed for that conditional-mean family. With ideal extraction, I.2.5–I.2.6 establish the parametric estimation rate, and I.2.7 gives sufficient sample-size conditions for overall risk improvement. In the experiments, generated-text quality is varied through generator capacity (Section 5.4, Table 3(c)): with the 3B generator, SEAR returns to the image-only range, in line with a smaller bias gain under larger generation noise.
-
No. The independence statement does not apply to image-generated descriptions. The revised Section 4.1 explicitly models generated structure as dependent on the image factors through . Human task knowledge in the prompt guides pretrained visual–language knowledge to select and express attributes of the same image. The structural view is thus a knowledge-guided expression of image content.
-
Assumption 2 concerns information retained conditional on the frozen image representation, not on the image itself. Attributes already present in the image can be only partly captured by that compressed representation yet accessible through a generated description. Their common source is therefore compatible with complementary predictive information: the generator and image encoder can express different task-relevant aspects of the same image. Section 4.2, Theorem 1 quantifies this contribution through conditional projection, and Appendix I constructs joint models in which observing the omitted, target-relevant structure strictly reduces representation bias.
Response to Reviewer 1WaW - Weakness 2 and Questions 3-5 - Cats vs Dogs statistics and evaluation
Thank you for recognizing the value of generated descriptions and for your detailed suggestions on prompt controls, statistical reporting, and the theoretical interpretation. We address these concerns through controlled comparisons, clarified evaluation, and explicit statistical models.
Weakness 2
Question 3
Question 4
Question 5
Response
Appendix C.6, Table 8 corresponds to Table 6 in the original submission. We have rechecked the two Cats vs Dogs entries and corrected them in the revised table. Each baseline run evaluates all 100 Cats vs Dogs test images, using the split in Appendix B, Table 5 and seeds 42, 43, and 44. The reported values are the arithmetic mean and the population standard deviation (variance denominator 3) over these three runs.
In seed order 42/43/44, CLIP Image scores 99.00/100.00/99.00, giving 99.33 ± 0.47; ViT Image scores 99.00/100.00/98.00, giving 99.00 ± 0.82. All values are F1-macro percentages; the two summaries are those reported in the revised Table 8.
The evaluation protocol is the same: the SEAR results in Table 1 and CLIP/ViT image-baseline results in Table 8 share the complete test split, F1-macro metric, and argmax prediction rule. Their training losses differ: SEAR uses cross-entropy for this single-label task, whereas the CLIP Image and ViT Image baselines use binary cross-entropy.
Table R3.3. Cats vs Dogs individual runs underlying the summaries in manuscript Appendix C.6, Table 8.
Note. F1-macro (%) on all 100 test images in each run. The mean is the arithmetic average over the three runs; SD is the population standard deviation with variance denominator 3. Table 8 reports the aggregate values.