- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies how to convert expert free-text opinions attached to photographs into supervision for vision-language models that generate structured defect reports, using private housing-condition reports from England. Since each opinion mixes visible evidence with an inferred cause that cannot be verified from the image, the authors argue that training on the full text would teach the model to assert unseen causes. They use a local text-only LLM to convert only the evidence part into a structured schema, filter the records with validation rules, and fine-tune three open-weight VLMs with QLoRA under a leakage-safe property-level split. Fine-tuning clearly outperforms prompting baselines in recognition and schema validity, while the choice of backbone matters little. An ablation further shows that labels converted from the full opinion leave recognition largely unchanged but make the model assert a cause on almost every image.
Critical strengths
- The paper targets a meaningful problem. Turning legacy expert prose into supervision is practically relevant to many domains.
- The evaluation protocol is careful for an applied paper, with many aspects considered.
Critical weaknesses
- The central finding of the label-content ablation appears to follow directly from the definition of supervised fine-tuning, and it is unclear what additional insight it provides. In Table 4, 90.9% of the full-opinion labels contain cause terms, and the corresponding model asserts a cause on 95.6% of test images, whereas 0.3% of the evidence-only labels contain cause terms, and that model does so on 0.0%. Supervised fine-tuning is trained precisely to reproduce the distribution of its targets, so two different target styles producing two matching output styles is the expected outcome, and it is difficult to imagine a result of this experiment that would have been surprising. The title and abstract frame this as a general principle ("what the label admits, the model asserts"), but the experiment does not go beyond confirming that a model imitates its labels. A more informative version of this study would examine cases in which the expected imitation does not hold. For instance, the training system prompt explicitly instructs the model to "never assert hidden causes" (Sec. 4), yet the full-opinion model asserts causes on almost every image, which suggests that the training targets fully override the instruction. Quantifying when and how strongly label content dominates explicit instructions, or how much cause-bearing label content is needed before the behaviour appears, would offer considerably more insight than the current two-point comparison.
- The methodological novelty is limited. The pipeline combines LLM-based annotation, rule-based filtering, QLoRA fine-tuning, and standard set-matching F1, all of which are established components, and the evidence-only constraint that the paper presents as its key design choice is implemented as a single instruction in the conversion prompt (rule 2 in Appendix A). The authors note in Sec. 2 that this rule is the findings/impression split from radiology report labelling, applied at conversion time, and they position the novelty in ablating it rather than in the rule. The careful leakage control and evaluation protocol are valuable, but they reflect good experimental practice and do not constitute a novel methodological contribution. Therefore, the paper is more like a well-executed engineering study of one private dataset, and in its current form it may be better suited to an applied or domain-specific venue than to ICLR.
- The writing of this paper deserves a separate comment, since it considerably harms readability. Reading this paper often feels like working through an exercise from a plain-English style guide in the spirit of Strunk and White, and only occasionally like reading an ICLR submission. The prose is short and declarative, punchlines usually land after a colon, and the four finding headers in the introduction read as slogans ("Adaptation, not prompting, delivers the task", "The backbone is second-order", "Label content sets faithfulness, not recognition", "Failure follows the supervision"). The central claim is then restated as a near-chiasmus in the title, the abstract, Sec. 5.4 and the conclusion. The text also abounds in unsupported absolute statements ("Every clinical, industrial, insurance and legal domain has accumulated the same asset", "appears exactly where the converted corpus is thinnest"), placed beside equally unexplained hedges ("in principle", "almost only", "reliably but modestly"). Most distracting is the amount of defensive writing aimed at objections that nobody has raised. The paper repeatedly tells the reader what things are not: the adaptation gain "is not a large-data artefact" (Sec. 5.5), the leakage "is removed on principle" (Sec. 3.5), the no-defect failure reflects missing negatives "rather than a backbone-specific failure" (Sec. 5.6) and "rather than model choice" (Sec. 7), and the dataset "is not the contribution" (Sec. 1). Honestly, after the third or fourth of these, the natural reaction in my mind is: who asked that? Several sentences go a step further and proactively deny a problem before anyone has suggested it, such as the reassurance that an earlier, leakier version of the split "had not inflated the earlier estimates" (Sec. 3.5), a version the reader never saw, or the claim that "none of this is specific to housing" (Sec. 6), which is asserted without a single experiment outside housing, and these mostly leave the reader wondering why the authors are so eager to distance themselves from every conceivable criticism, even at the clear expense of readability. As a result, a fairly simple set of experiments is wrapped in rhetoric that the reader has to peel away sentence by sentence before reaching the actual claims and numbers. I believe this paper's writing should be significantly improved for better readability, if it wants to be read and fairly evaluated by a human reviewer. I would strongly suggest that the authors rewrite it themselves.
- The paper also does not seem to follow the ICLR template, as the body font is not the default Times.
Questions for authors
- Given that supervised fine-tuning is trained to reproduce its targets, what outcome of the label-content ablation would have contradicted your central claim?
- The full-opinion model asserts causes on 95.6% of images, although the system prompt requires the model "never assert hidden causes". How does this behaviour change as the portion of cause-bearing labels varies?
- Beyond ablating rule 2, what methodological contribution distinguishes this pipeline from prior LLM-based labelling of radiology reports?