acceptodds
Under review as a conference paper at ICLR 2027

What the Label Admits, the Model Asserts: Evidence-Constrained Supervision of Vision-Language Models from Expert Prose

Abstract

Professional inspection practices hold years of photographs paired with expert free-text opinions written for human readers, not for model supervision. We study how to turn such legacy prose into supervision for a vision-language model (VLM) that produces structured, evidence-constrained defect reports, using a private corpus of 4,231 photographs from 586 expert-witness housing-condition reports prepared for disrepair claims in England. Each surveyor opinion follows an evidence–cause–remedy convention; a locally hosted text-only LLM converts only the visible-evidence component into a controlled JSON schema over a 52-type, 20-family defect taxonomy derived from the corpus, so that diagnosed causes never become visual labels, and every record passes structural, taxonomic and semantic validation, with flagged records excluded from training. Three open-weight VLMs of 7–9B parameters (Qwen2.5-VL, Qwen3-VL, GLM-4.1V) are adapted with QLoRA on a single 24 GB GPU under a property-level, near-duplicate-safe split. Adaptation raises type-level micro-F1 from at best 0.155 (few-shot prompting) to 0.432 with disjoint bootstrap intervals, and schema-valid output from 69.6% to 100%, while preserving the expert's evidence-only register. Under five-fold property-level cross-validation with a paired property-level bootstrap, swapping among three backbones from two vendors moves micro-F1 by at most 0.03, an order of magnitude less than adaptation itself, and the newest backbone lowers the no-defect false-positive rate from 56.0% to 44.0%. A label-content ablation then separates recognition from faithfulness: converting the whole opinion instead of only its evidence leaves recognition unchanged within bootstrap noise, but the model reproduces the register of its labels almost exactly, asserting an unverifiable cause on 95.6% of images against 0.0% for the evidence-only model. Recognition is set by adaptation and data volume (a training-set-size ablation shows no saturation); what the model asserts is set by what the label admits. We release the schema, taxonomy, prompts, validation rules and evaluation code; the imagery remains private.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
vision-language models, domain adaptation, parameter-efficient fine-tuning, structured report generation, supervision from expert text, label design, building inspection, hallucination

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_fze2OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how to convert expert free-text opinions attached to photographs into supervision for vision-language models that generate structured defect reports, using private housing-condition reports from England. Since each opinion mixes visible evidence with an inferred cause that cannot be verified from the image, the authors argue that training on the full text would teach the model to assert unseen causes. They use a local text-only LLM to convert only the evidence part into a structured schema, filter the records with validation rules, and fine-tune three open-weight VLMs with QLoRA under a leakage-safe property-level split. Fine-tuning clearly outperforms prompting baselines in recognition and schema validity, while the choice of backbone matters little. An ablation further shows that labels converted from the full opinion leave recognition largely unchanged but make the model assert a cause on almost every image.

Critical strengths
  • The paper targets a meaningful problem. Turning legacy expert prose into supervision is practically relevant to many domains.
  • The evaluation protocol is careful for an applied paper, with many aspects considered.
Critical weaknesses
  • The central finding of the label-content ablation appears to follow directly from the definition of supervised fine-tuning, and it is unclear what additional insight it provides. In Table 4, 90.9% of the full-opinion labels contain cause terms, and the corresponding model asserts a cause on 95.6% of test images, whereas 0.3% of the evidence-only labels contain cause terms, and that model does so on 0.0%. Supervised fine-tuning is trained precisely to reproduce the distribution of its targets, so two different target styles producing two matching output styles is the expected outcome, and it is difficult to imagine a result of this experiment that would have been surprising. The title and abstract frame this as a general principle ("what the label admits, the model asserts"), but the experiment does not go beyond confirming that a model imitates its labels. A more informative version of this study would examine cases in which the expected imitation does not hold. For instance, the training system prompt explicitly instructs the model to "never assert hidden causes" (Sec. 4), yet the full-opinion model asserts causes on almost every image, which suggests that the training targets fully override the instruction. Quantifying when and how strongly label content dominates explicit instructions, or how much cause-bearing label content is needed before the behaviour appears, would offer considerably more insight than the current two-point comparison.
  • The methodological novelty is limited. The pipeline combines LLM-based annotation, rule-based filtering, QLoRA fine-tuning, and standard set-matching F1, all of which are established components, and the evidence-only constraint that the paper presents as its key design choice is implemented as a single instruction in the conversion prompt (rule 2 in Appendix A). The authors note in Sec. 2 that this rule is the findings/impression split from radiology report labelling, applied at conversion time, and they position the novelty in ablating it rather than in the rule. The careful leakage control and evaluation protocol are valuable, but they reflect good experimental practice and do not constitute a novel methodological contribution. Therefore, the paper is more like a well-executed engineering study of one private dataset, and in its current form it may be better suited to an applied or domain-specific venue than to ICLR.
  • The writing of this paper deserves a separate comment, since it considerably harms readability. Reading this paper often feels like working through an exercise from a plain-English style guide in the spirit of Strunk and White, and only occasionally like reading an ICLR submission. The prose is short and declarative, punchlines usually land after a colon, and the four finding headers in the introduction read as slogans ("Adaptation, not prompting, delivers the task", "The backbone is second-order", "Label content sets faithfulness, not recognition", "Failure follows the supervision"). The central claim is then restated as a near-chiasmus in the title, the abstract, Sec. 5.4 and the conclusion. The text also abounds in unsupported absolute statements ("Every clinical, industrial, insurance and legal domain has accumulated the same asset", "appears exactly where the converted corpus is thinnest"), placed beside equally unexplained hedges ("in principle", "almost only", "reliably but modestly"). Most distracting is the amount of defensive writing aimed at objections that nobody has raised. The paper repeatedly tells the reader what things are not: the adaptation gain "is not a large-data artefact" (Sec. 5.5), the leakage "is removed on principle" (Sec. 3.5), the no-defect failure reflects missing negatives "rather than a backbone-specific failure" (Sec. 5.6) and "rather than model choice" (Sec. 7), and the dataset "is not the contribution" (Sec. 1). Honestly, after the third or fourth of these, the natural reaction in my mind is: who asked that? Several sentences go a step further and proactively deny a problem before anyone has suggested it, such as the reassurance that an earlier, leakier version of the split "had not inflated the earlier estimates" (Sec. 3.5), a version the reader never saw, or the claim that "none of this is specific to housing" (Sec. 6), which is asserted without a single experiment outside housing, and these mostly leave the reader wondering why the authors are so eager to distance themselves from every conceivable criticism, even at the clear expense of readability. As a result, a fairly simple set of experiments is wrapped in rhetoric that the reader has to peel away sentence by sentence before reaching the actual claims and numbers. I believe this paper's writing should be significantly improved for better readability, if it wants to be read and fairly evaluated by a human reviewer. I would strongly suggest that the authors rewrite it themselves.
  • The paper also does not seem to follow the ICLR template, as the body font is not the default Times.
Questions for authors
  • Given that supervised fine-tuning is trained to reproduce its targets, what outcome of the label-content ablation would have contradicted your central claim?
  • The full-opinion model asserts causes on 95.6% of images, although the system prompt requires the model "never assert hidden causes". How does this behaviour change as the portion of cause-bearing labels varies?
  • Beyond ablating rule 2, what methodological contribution distinguishes this pipeline from prior LLM-based labelling of radiology reports?
Reviewer_GQH3OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper introudces a corpus of data for training VLMs to produce defect reports. It trains three models on the data, and shows improvements in F1 on prediting the type of defect. The paper only releases some of the data.

Critical strengths

Hard to find any -- the data could be useful for a narrow domain if released, but the photographs are not released, and thus the dataset is not useable by others.

Critical weaknesses
  • The paper is extremely hard to understand and shows all the standard signs of LLM overuse. The opening statement is patently false: "Every clinical, industrial, insurance and legal domain has accumulated the same asset: images attached to expert free text that was written to persuade or inform a human reader." There are lots of domains which do not have images. This kind of oversight raises questions regarding to what degree the authors inspected the output of the model before submitted. Another major oversight that would have been caught by even a cursory look at the paper: the paper doesn't follow the template and is in the wrong font.

  • It's not clear what the contribution of the paper is. The claimed contribution is a recipe for rewriting existing data and creating data splits, which consists of standard choices and tools. The dataset is not released and therefore not claimed as a contribution.

Questions for authors

NA

Reviewer_r9dkOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The study converts the expert-witness surveyor's opinions into structured defect labels using a local large language model, retaining only the portion of each opinion that relates to visible evidence, and then finetunes three VLMs with QLoRA. It finds that (1) finetuning performs better than prompting; (2) the choice of backbone has little effect; (3) the content of the labels determines whether the model includes causes but does not affect recognition; and (4) false positives on images of undamaged objects are due to the lack of negative examples.

Critical strengths

A careful evaluation was carried out, involving property level splits, the removal of near duplicates across reports, paired property level bootstrap analysis, and a sensitivity check for flagged records; the figures are not likely to have been inflated due to leakage.

The difference in the stated causes can be attributed to the label text since 3,086 of the 3,219 shared training images have the same label sets in both groups and the label-content ablation is well controlled.

Critical weaknesses

The contribution is small and mostly expected; the fact that finetuning outperforms zero-shot prompting on a private taxonomy, and that a finetuned model matches the register of its targets, can both be understood from how SFT functions; the label-content result is very similar to that of Ramesh et al. (ML4H 2022), who removed references to previous exams from the targets of radiology reports. The authors present this as evidence of faithfulness, but the metric only takes into account cause terms. The output's connection to the image is never assessed, and even the evidence-only model still shows defects on 56% of images that have no defects. The 2.3x improvement over prompting is also overstated: the prompted baselines lose at least 30% of their outputs to schema errors, since they employ no constrained decoding and no in-corpus few-shot examples.

The main claims cannot be verified since the training and test labels are derived from the same LLM conversion process, the two extractors agree on only 56.2% when assessed at the family level, expert review has not taken place, and since one opinion can relate to several photos it is possible for a label to identify defects that are not visible in that particular image. F1 thus measures the agreement with the converter rather than defect recognition.

Questions for authors

In a sample of test images which has been examined by humans, what fraction of the reference problems are actually visible in the image?

If the two figures are different, then which of the extractors' labels is adopted? Furthermore, section 5.6 says that 15 issues have been produced, but Appendix E indicates 14 (that is, 0.56 times 25); and the '25%/50%' subsets correspond to 37%/54% of the training reports.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.