acceptodds
Under review as a conference paper at ICLR 2027

Evaluating Vision-Language Models on Images of Individuals with Limb Deficiencies

Abstract

Strong performance on general benchmarks does not establish whether vision- language models (VLMs) correctly interpret images of individuals with limb deficiencies. For this population, applications such as preparing visual evidence for Paralympic athlete classification depend on correctly identifying residual limbs, prosthetic devices, and body-device configurations. We find that VLMs can miss this evidence or assign it to the wrong body regions, so later reasoning begins from incorrect visual information. To examine these failures, we introduce INCLUSIVEVLM-LIMBEVIDENCE, a test-only benchmark on about 17k open- world images of this population. It provides over 52k direct evidence queries and over 20k compatibility instances, with human reference results and prompt paraphrases. The design separates recognizing evidence from using it in struc- tured decisions. LIMBEVIDENCE-PRESENCE tests evidence detection. Because detection alone does not establish where evidence belongs, LIMBEVIDENCE- ATTRIBUTION tests body-region assignment in parallel Constrained and FreeForm formats. To examine whether models can also use this evidence in structured deci- sions, LIMBEVIDENCE-COMPATIBILITY tests option matching under benchmark- defined relations through parallel Category and Diversity probes. Across widely used general-purpose VLMs, models often detect limb evidence but fail to iden- tify its body region or side. Partial option matches often fall short of complete compatible-set recovery. Models can also repeat the same incorrect answer across paraphrases. The larger model in our scale comparison improves coarse detection but does not resolve attribution and set-recovery errors. These results reveal a gap between recognizing limb evidence and using its spatial and relational details correctly. They also show that response consistency cannot replace correctness. INCLUSIVEVLM-LIMBEVIDENCE makes these overlooked capability limits mea- surable and supports research toward visual AI that better serves individuals with limb deficiencies.

Primary area:
datasets and benchmarks
Keywords:
Individuals with Limb Deficiencies, Benchmark

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_27ZkOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper focuses on a new field, the images of individuals with limb deficiencies. The problem is interesting and reasonable. Based on the problem, the paper introduces InclusiveVLM-LimbEvidence, a benchmark for evaluating vision-language models on images of individuals with limb loss in real-world settings. The authors incorporate new annotations for residual limbs and prosthetics, structuring the evaluation around presence detection, body part attribution, and matching with candidate device categories, while assessing several mainstream VLMs. Results indicate that coarse-grained recognition capabilities do not guarantee accurate identification of body parts and laterality, nor do they ensure the correct retrieval of the full set of matching options. Furthermore, consistent responses to rephrased queries may still be incorrect.

Critical strengths
  1. The problem setting is interesting and useful.
  2. Separating evidence detection, spatial attribution, and structured matching helps failures analysis, which is good for diagnostic evaluation.
  3. The new annotations extend the source dataset meaningfully and the experiments reveal some useful preliminary results.
  4. The failure analysis for current SOTA VLM is beneficial.
Critical weaknesses
  1. In Appendix C9, the experiment results show that constant predictions achieve 65.79% Presence accuracy, predicting prosthetic-only, and always predicting an empty set yields 14.67% Diversity Exact. This should be compared with primary results and more analysis is needed to distinguish the true performance from label priors.
  2. Errors from visual perception have not yet been fully distinguished from those caused by prediction formatting or task interpretation. On the same set of 500 samples, the Attribution mIoU for Qwen3-VL-4B-Thinking improved from approximately 17.2% to 43.1% when using a free-text format. This significant difference indicates that the label expression affects the performance substantially.
Questions for authors
  1. How do model results compare with simple baselines across all primary metrics, particularly when empty and nonempty targets are evaluated separately?
  2. How much attribution error remains under a common, natural-language output interface?
  3. Can the authors provide independent annotation-agreement statistics and executed option-permutation controls?
4 comments
AuthorsOpenReview ↗

Response to Reviewer 27Zk

Thank you for recognizing the benchmark’s value and the usefulness of its diagnostic decomposition.

1. The baseline comparisons reveal limitations in current models; they are not omitted from the interpretation.

Appendix C.9 reports the 65.79% Presence majority baseline and 14.67% always-empty Diversity Exact baseline. All models use the same task-specific test inputs, so these references apply to their shared pools. Only GPT-5 (77.2%) and Qwen3-VL-235B (74.23%) exceed the Presence majority baseline, and all model Exact scores fall below the always-empty reference. We do not treat the remaining scores as evidence of above-prior recognition or set recovery.

The requested empty/nonempty distinction is already partly available. Table 7(c) and Appendix C.9 give the 235B model’s Exact as 210/10,765 = 1.95% on nonempty targets and 177/1,851 = 9.56% on empty targets. Poor recovery therefore occurs within both groups, rather than resulting solely from their mixture. A complete baseline panel across all primary metrics is not reported.

2. Substantial attribution mismatch remains under natural-language output.

Section 4.1, Table 4(b), and Appendix C.5 directly answer this question. On the same 500 image–target pairs, Constrained→FreeForm mIoU is 17.22→43.1 for Qwen3-VL-4B-Thinking, 15.87→44.7 for Qwen3-VL-8B-Thinking, and 26.51→30.5 for GPT-5. Parsing succeeds for 467/500, 452/500, and 417/500 responses, respectively.

These results establish a format effect, while also showing that natural-language output does not resolve attribution. They do not establish that every remaining mismatch arises exclusively from visual perception. Appendix D.2 explicitly distinguishes observed task errors from such a causal interpretation.

3. Annotation review and construction-time randomization are documented procedures, not unreported experiments.

Section 3.3 and Appendix B.4 describe second-annotator review, re-annotation, and classifier-supported adjudication. Appendix B.6 documents randomized option order during construction.

Independent annotation-agreement coefficients and executed repeated-permutation results are not reported. We do not present adjudication as an independent agreement measurement, or construction-time randomization as demonstrated permutation robustness. The existing evidence supports reviewed targets and the reported model behavior under the fixed evaluation protocol.

Thanks for clarifying the FreeForm results. The comparison on the same 500 examples answers my question about how much attribution error remains when models can respond in their own words.

My main concern is still how much the models gain from seeing the image. Always predicting “prosthetic only” already gives 65.79% on Presence. I’d find the results easier to interpret with simple baselines evaluated on the same examples as each model, and with a comparison of model performance with and without the image. I’d also like to know how often annotators agreed before adjudication and whether models change their answers when the options are reordered. I keep my original rating.

AuthorsOpenReview ↗

Follow-up to Reviewer 27Zk

Thank you for confirming that the comparison on the same 500 examples resolves your question about attribution under natural-language output.

Presence baselines on the prompts actually scored. For Qwen3-VL-235B, the comparison is exact because all 26,406 responses were valid. The model answered 19,601 prompts correctly, while always predicting “prosthetic only” answers 17,373 correctly. Table 6(b) shows where the difference comes from. The model correctly answered 4,388 prompts from the two minority classes and gave up 2,160 correct answers on the majority class, for a net gain of 2,228 correct answers, or 8.44 percentage points, on identical prompts.

Qwen3-VL-30B-A3B-Instruct achieved a parsing success rate above 90% across the 26,406 Presence prompts. Its accuracy on successfully parsed responses was 23.9%. Even in the worst case for the majority baseline, if every parsing failure occurred on a “prosthetic only” prompt, always predicting “prosthetic only” would still achieve at least 61.99% accuracy on the exact prompts scored for the 30B model (Table 6(b)). Thus, even under the assumption most favorable to the model, it falls more than 38 percentage points below the majority baseline.

Image use, annotation agreement, and option order. Quantifying the gain from seeing the image requires running the same model on the same prompts with and without the image. We have not reported that experiment. Appendix B.4 documents second-annotator review and adjudication, but the submission gives no agreement rate before adjudication. Appendix B.6 documents randomized option order during grid construction, but the same grids were not evaluated under repeated permutations, so we cannot report how often reordering changes a prediction.

Could you clarify whether you dispute the observed body-region and option-selection errors, or whether your recommendation turns on the further question of what causes those errors? The former can be checked against the recorded predictions and targets. The image-removal and permutation experiments would address the latter.

Thanks for clarifying. I agree that the 235B model beats the majority baseline on Presence and gets the evidence type right more often than the body region. I also accept the body-region and option-selection errors you report. My concern is how much seeing the image helps the model identify the evidence in the first place. Since always answering “prosthetic only” already gives 65.79% accuracy, I’d like to see how the same model does on the same prompts without the image before drawing the broader conclusion that models “often detect limb evidence.”

Reviewer_z3pCOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This test-only benchmark asks whether general-purpose VLMs can reason about residual limbs and prostheses in images of people with limb deficiencies.

It reuses 16,850 LDPose images and adds pixel-level masks, body-region labels, and body–device configuration records. There are three probe families:

  • Presence: four-way detection.
  • Attribution: 8 limb segments × 2 evidence types, in Constrained and FreeForm formats.
  • Compatibility: Category hit@1 and Diversity set recovery.

Across 21 checkpoints from six families plus a five-rater human reference, the authors conclude that models detect limb evidence but get its region or side wrong, and that scaling Qwen3-VL from 30B to 235B doesn't fix this.

Critical strengths

An important, under-served target, backed by a real annotation investment. People with limb deficiencies are almost absent from VLM benchmarks. The new masks, region labels and configuration records on 16,850 images, guided and adjudicated by an IPC-certified classifier, are a genuine contribution. The data governance pieces (datasheet, licensing, DUA, takedown channel) are good to see.

A sensible decomposition with careful bookkeeping. Splitting the problem into detection, region/side attribution and matching asks a sharper question than "does the model understand disability images." The 235B diagnostics are internally consistent wherever I checked (e.g., 16,850 = 8,802 + 8,048; 19,601/26,406 = 74.23%).

Critical weaknesses

I thank the authors for the clarification. All models receive the same input instances, so I withdraw my earlier concern about different test sets. The parsing rate of the 30B model (above 90%) also rules out the template explanation I suggested.

My main concerns are about how the scores can be interpreted.

1. Most models are not clearly above trivial answer priors.

  • On Presence, always answering "b" gives 65.79% (C.9). Of the 21 models in Table 1, only GPT-5 (77.2) and Qwen3-VL-235B (74.2) exceed it.
  • For 235B, the +8.44-point margin and the gap between evidence type (74.40%) and region/side (40.11% / 40.59%) do support the "detect but mis-localize" pattern. For most other models, detection itself is not yet established.
  • On Diversity, "select nothing" gives 14.67% Exact, above every model and the human reference. My rough select-all estimate (about 61% F1) could not be checked because the grid-size distribution is not reported.

The rebuttal confirms that a complete baseline panel, grid-size statistics and an image-removal (blind) control are not reported. These would make the main claims much easier to interpret.

2. Comparisons across models use different retained subsets. Because parsing failures are excluded rather than scored, each model is evaluated on its own retained subset. The bolded "best" entries and the 30B → 235B comparison are therefore not fully like-for-like. Format also matters a lot: in the matched Table 4(b), Constrained → FreeForm mIoU goes 17.22 → 43.1 and 15.87 → 44.7 for the two Qwen3-VL Thinking models. So part of the "attribution failure" reflects the 16-label format.

Questions for authors
  1. Could you report, per task and model, the number of retained items, and either score parsing failures as incorrect or compare models on a common retained subset?
  2. Could you add a full baseline panel for each task (majority class for Presence; select-all, select-none and cardinality-matched random for Diversity; chance for Category), together with the grid-size distribution and an image-removal control? If select-all turns out to be competitive on Diversity F1, would a different primary metric be more informative?
  3. Since the pool has no "neither" targets, could the main text present Presence as effectively three-way and report the three-class macro-F1 (58.41 for 235B) alongside the four-label one?
  4. Could the main text discuss the format effect from Table 4(b) more prominently when interpreting the attribution results?
4 comments
AuthorsOpenReview ↗

Response to Reviewer z3pC (1/2)

Thank you for the detailed review and for recognizing the annotation contribution.

We regret that the overall assessment gives insufficient weight to the contribution that the review itself explicitly recognizes. The review describes our annotations as a “genuine contribution” and the evaluation decomposition as “sensible.” Our benchmark makes specific failures on an underserved population measurable. Poor performance by the evaluated models does not, by itself, invalidate that contribution. Moreover, the concern about different input test sets rests on an incorrect inference. The soundness assessment should distinguish demonstrated limitations of the benchmark from failures of the models being evaluated, and should be based on the evaluation protocol we actually used. We address these points directly below.

1. All models receive the same test images and questions.

The inference that models were given different test sets is incorrect. Within each task, all models receive identical input instances. Table 3 and Appendix B.2 report the task sizes:

  • Presence and Constrained Attribution: each uses 26,406 questions over 8,802 images.
  • Category: 8,048 questions over 8,048 images.
  • Diversity: 12,616 questions over 8,048 images.
  • FreeForm: the same 500 image–target pairs.

The approximately 2,600 and 145 counts inferred for GPT-5 and DeepSeek-VL2-27B do not describe the inputs supplied to these models. Appendix C.2 documents the bootstrap procedures. Confidence-interval widths are not substitutes for the actual input counts.

Parsing failures are excluded from scoring. Consequently, models share the input pool, while their retained scoring subsets can differ. We distinguish these quantities rather than claiming that identical inputs necessarily produce identical valid-response counts.

2. The 30B model’s parsing success rate exceeds 90%; its low accuracy is not a penalty for unparseable answers.

For Qwen3-VL-30B-A3B-Instruct, parsing success exceeds 90%. Its 23.9% Presence accuracy is calculated over successfully parsed responses. Parsing failures are skipped, not counted as incorrect answers. The reported score therefore measures correctness among validly parsed predictions.

Even if every skipped response were generously credited as correct, the 30B model’s accuracy over the complete input pool would remain below 32%, far below the 235B model’s 74.23%. This is an upper-bound calculation, not an additional experiment. Excluding parsing failures cannot account for the detection gap between these checkpoints. The comparison concerns the reported checkpoints and settings, rather than parameter count in isolation (Appendix C.4).

Parsing statistics are also reported in the submission. Table 4(b) gives FreeForm coverage of 467/500 (93.4%), 452/500 (90.4%), and 417/500 (83.4%) for Qwen3-VL-4B-Thinking, Qwen3-VL-8B-Thinking, and GPT-5. Appendix C.6 reports 100% output coverage and zero parsing/protocol errors for the 235B Presence and Constrained Attribution evaluations. Its Constrained Attribution mIoU remains 24.76% despite fully valid outputs.

3. Output-format sensitivity is already an explicit result, and the paired comparison must use matched items.

Table 1 places full-pool Constrained results alongside FreeForm results on 500 items. Subtracting these columns does not produce paired format-effect estimates for all 21 models.

The matched comparison is Table 4(b), using identical 500-item inputs and targets within each model. Constrained→FreeForm mIoU is 17.22→43.1, 15.87→44.7, and 26.51→30.5 for the two Qwen3-VL Thinking models and GPT-5, respectively. Section 4.1 explicitly identifies this format effect. Natural-language output improves some results substantially, but does not resolve attribution. High parsing coverage does not imply that format or label interpretation has no effect. These are separate questions.

AuthorsOpenReview ↗

Response to Reviewer z3pC (2/2)

4. Prior baselines constrain the interpretation of the scores.

Appendix C.9 reports the 65.79% Presence-majority and 14.67% always-empty Diversity Exact references for the complete input pools. Since invalid responses are excluded, an exact comparison for another model must apply the same retained-item subset to both that model and its baseline. The full-pool references also cannot automatically be transferred to the separately sampled human assignments in Appendix C.3.

For 235B Presence, where coverage is 100%, 74.23% exceeds the majority baseline by 8.44 percentage points. The recognition–attribution distinction is additionally supported within the same Attribution responses: evidence-type correctness is 74.40%, versus 40.11% for body region and 40.59% for side (Table 6(c)). This is more specific than claiming that every evaluated model demonstrates above-prior detection.

For Diversity, Appendix C.9 explicitly cautions against interpreting aggregate Exact alone as evidence of image use. The 235B outcome analysis gives 1.95% Exact on nonempty targets and 9.56% on empty targets. Poor recovery persists within both groups.

The estimated select-all F1 of approximately 61% is not established by the reported aggregate statistics. Its exact sample-average value is

1N∑i2∣Yi∣∣Oi∣+∣Yi∣,\frac{1}{N}\sum_i \frac{2|Y_i|}{|O_i|+|Y_i|},

which requires each item’s target and option counts. The mean target size and an eight-option illustration are insufficient. Category chance likewise depends on the accepted answers and available choices, rather than automatically being 25% (Appendix B.6). A complete baseline panel, grid-size distribution, and image-removal control are not reported.

F1 measures set overlap; it is not standalone evidence of visual reasoning. The paper therefore reports it alongside precision, recall, Exact, Partial, prediction sizes, and empty-target outcomes.

5. The four-label answer space and the three populated target classes are explicitly distinguished.

Appendix B.5, Table 6(b), and Appendix F disclose that the pool has no “neither” targets. The four-label macro-F1 is 43.81%, while averaging the three supported classes gives 58.41%. This changes the averaging inventory, not the predictions or primary accuracy. Correct rejection on images with neither evidence type is outside this pool’s measured coverage.

6. Ethics approval, image rights, and human-study documentation are distinct.

Section 3.2 and Appendices E.1–E.2 document institutional review for secondary image use, controlled access, redistribution restrictions, and a correction/removal pathway. The paper does not equate ethics approval with copyright permission, or face blurring with guaranteed anonymity. Appendix C.3 describes the raters and study procedures.

The requested figure-specific permission records and recruitment/compensation details are not enumerated in the submission. Those specific records cannot be established merely by citing the general ethics statement.

Reviewer_z3pCOpenReview ↗

Revised comment after rebuttal

I thank the authors for the detailed clarification. I accept that all models receive the same input instances and that parsing failures are excluded rather than counted as errors, and I withdraw my concern about different input sets.

However, my core concerns remain.

Evaluation. Because parsing failures are excluded, models are scored on different retained subsets. The response also confirms that a complete baseline panel, a blind or image-removal control, and grid-size statistics are not reported, and the strongest model exceeds the Presence majority baseline by only 8.44 points.

Ethics. The authors acknowledge that figure-specific permission records and rater recruitment and compensation details are not provided. For a dataset of images of people with limb differences, I consider these necessary.

I keep my score of 1 and my ethics flag.

AuthorsOpenReview ↗

Follow-up to Reviewer z3pC

Thank you for clarifying your remaining concerns and for withdrawing the concern about different input sets.

Recruitment and compensation. We recruited all five raters from among our laboratory colleagues. None received payment for participating in the evaluation. Two specialize in sports rehabilitation for people with disabilities. One is a postdoctoral research fellow and the other is a third-year PhD candidate. The other three are postdoctoral researchers with computer science backgrounds. Each rater completed 200 items, yielding 1,000 item evaluations in total (Appendix C.3).

Ethics approval and the photographs in Figures 3 and 4. Appendix E.1 reports approval by an institutional Human Research Ethics Committee and specifies the scope of the approved protocol, including secondary use of source photographs, annotation, secure storage, and controlled access. We withheld the institution’s name and approval identifier to preserve author anonymity. If the ethics reviewers require documentary verification of approval, we ask the Area Chair to indicate the appropriate confidential channel for a redacted record. The publication basis for the photographs in Figures 3 and 4 is a separate question. Appendix E.2 states that the source photographs retain their original rights. The submission does not document the publication basis for each photograph individually.

The 30B–235B Presence gap exceeds 47 percentage points even on the same scored prompts. Qwen3-VL-30B-A3B-Instruct successfully parsed more than 90% of the 26,406 Presence prompts. Its accuracy on those retained prompts was 23.9%, and at most 2,640 prompts were excluded. The 235B model returned valid answers on all 26,406 prompts and answered 19,601 correctly (Table 6(b), Appendix C.6). Even under the assumption most favorable to the reviewer’s concern, removing 2,640 prompts that 235B answered correctly would leave it with 16,961 correct answers out of 23,766, or at least 71.36%. Differences in parsing coverage therefore cannot explain the gap between these checkpoints.

The 8.44-point gain over the Presence majority baseline was measured on identical items. Table 6(b) shows exactly where it comes from. The 235B model correctly classified 4,388 of the 9,033 prompts that the constant “prosthetic only” answer misses. On the majority class, it made 2,160 errors that the constant answer avoids. The net gain was 2,228 additional correct answers across the same 26,406 prompts. Quantifying how much of this gain comes from seeing the image would require a matched image-removal control. The full baseline panel and grid-size distribution requested in the review are also unreported.

We ask you to reconsider the score in light of the common-item comparison and the recruitment and compensation details above. If the omission of those details from the original submission remains a principal reason for the score of 1 (clear rejection) after this clarification, we deeply regret the weight given to that omission when the review identifies no ethical breach in how the raters were recruited or compensated.

Reviewer_nDc1OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper introduces Inclusive VLM-LimbEvidence, a test-only benchmark using 16,850 open-world images to assess evidence detection, body-region attribution, and body-device compatibility. Evaluations across open- and closed-weight VLMs, deterministic metrics, paraphrase diagnostics, and human references show that models often detect limb evidence but mislocalize it or fail to recover complete compatible sets. Increased scale and response consistency do not reliably eliminate these errors.

Critical strengths
  1. The paper addresses an important and under-evaluated population. The decomposition from evidence presence to spatial attribution and relation-conditioned set recovery is conceptually useful because it exposes failures that aggregate recognition benchmarks would miss.
  2. The benchmark and diagnostic reporting are substantial. The submission includes new masks and body-region labels, evaluates a broad set of VLMs, reports human reference results, and distinguishes evidence-type, region, side, exact-set, partial-hit, and paraphrase-consistency behavior. The deterministic normalization and the authors' caution about metric interpretation improve transparency.
Critical weaknesses

1.The experiments do not sufficiently isolate visual understanding from dataset priors, prompt artifacts, or output-format effects. The paper lacks text-only or image-removal controls, shuffled-image tests, option permutations, left-right flips, and targeted occlusion studies. This is important because the Presence majority baseline reaches 65.79%, compared with 74.23% for Qwen3-VL-235B, while Diversity metrics may be influenced by trivial selection strategies. 2.The external validity of LimbEvidence-Compatibility is also unclear. Its body-part-to-device relations are benchmark-defined, but the paper provides limited evidence that Category and Diversity represent meaningful real-world requirements rather than artificial option matching. The human reference includes only five raters with partially different assignments, no defined group of people with limb deficiencies or professional prosthetics panel, and no overall agreement coefficient. These limitations weaken both the human-model comparison and the broader significance of the compatibility results.

Questions for authors

1.How are “insufficient information” cases distributed across Category and Diversity probes, and how often do models appropriately select this option relative to humans? 2.What safeguards were used to ensure that Compatibility grids do not introduce spurious visual cues (backgrounds, lighting, cropping conventions) that models could shortcut, and can you provide a hard-negative mining summary? 3.Could you include stratified analyses by limb segment (upper vs lower), side (left vs right), activity/setting, and occlusion level to help researchers target specific weaknesses? 4.Given you possess pixel-level masks, would you consider adding an auxiliary mask-level attribution score to complement set mIoU and better characterize partial localization errors? 5.Can the author report inter-annotator agreement (e.g., κ or percent agreement) for (a) residual/prosthetic masks, (b) derived body-region/side labels, and (c) accepted option sets for Compatibility?

1 comment
AuthorsOpenReview ↗

Response to Reviewer nDc1

Thank you for recognizing the importance of the population and the value of the diagnostic decomposition.

1. Output-format effects are already measured on matched inputs.

Section 4.1, Appendix C.5, and Table 4(b) compare Constrained and FreeForm on the same 500 image–target pairs. Attribution mIoU changes from 17.22% to 43.1% for Qwen3-VL-4B-Thinking, 15.87% to 44.7% for Qwen3-VL-8B-Thinking, and 26.51% to 30.5% for GPT-5. Their parsing coverage is 93.4%, 90.4%, and 83.4%, respectively.

Thus, format sensitivity is an explicit empirical finding, rather than an explanation omitted from the analysis. The remaining FreeForm target mismatch is also directly observable. Appendix D.2 distinguishes these output-level findings from a causal claim that every error originates in visual perception. Image-removal, shuffled-image, flip, occlusion, and repeated-permutation experiments would address further causal questions; we do not present the existing diagnostics as substitutes for those unreported interventions.

2. Compatibility evaluates a specified visual relation.

Section 3.4 and Appendix B.6 define correspondence between body-part categories in the image and device candidates. Applying this relation requires identifying the relevant categories and selecting the matching options. Appendix D.4 explicitly excludes clinical device suitability, rehabilitation decisions, and personalized recommendations. Clinical suitability is not the benchmark’s ground-truth definition.

The human study is a reference against these reviewed targets. Appendix C.3 and Table 4(a) report five raters, each completing 200 items: three with computer-science backgrounds and two with disability-research experience. Their group Diversity F1 values are 47.03% and 74.22%, respectively. The paper identifies their backgrounds and partially overlapping assignments; these results are descriptive references, not a controlled estimate of expertise or a professional performance ceiling.

3. Insufficient information and empty targets have different meanings.

Appendix B.6 explicitly distinguishes an insufficient-information response, an empty matching set, and a parsing failure. Table 7(c) analyzes empty-target selection errors, not uncertainty-based abstention. The requested insufficient-information frequencies and human–model abstention comparison are not reported.

Appendix B.6 also documents source-device crops, sampled distractors, and randomized option order. These are construction procedures. They do not establish robustness to background, lighting, cropping, or repeated option permutations, and the submission does not claim a completed hard-negative mining study.

4. Spatial breakdowns are available; mask scoring would require different predictions.

Table 6(d) already reports F1 by upper arm, forearm, thigh, calf, side, and evidence type for the 235B model. Activity/setting and occlusion strata are not reported.

Attribution predicts semantic label sets, as defined in Table 3 and Appendix B.3. The models do not predict pixel masks. Consequently, adding mask IoU would require a new output and evaluation protocol; it cannot be obtained simply by rescoring the current text responses.

5. Annotation review is documented, but it is not an independent agreement coefficient.

Section 3.3 and Appendix B.4 describe second-annotator review of every mask and derived label/option set, re-annotation, and adjudication supported by an IPC-certified classifier. This is the documented quality-control process. Independent agreement coefficients for the masks, derived labels, and accepted option sets are not reported, and we do not equate adjudicated agreement with independent annotation agreement.

Reviewer_enW2OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces InclusiveVLM-LimbEvidence, a benchmark for evaluating VLMs on images of individuals with limb deficiencies. Built upon LDPose, it provides fine-grained annotations and three evaluation tasks: Presence, Attribution, and Compatibility. Experiments across multiple VLMs reveal substantial limitations in recognizing and reasoning about residual limbs and prosthetic devices.

Critical strengths
  1. Meaningful research problem. The paper addresses an underrepresented population in VLM evaluation, highlighting important limitations in existing models.

  2. Comprehensive evaluation. The benchmark provides fine-grained annotations and evaluates multiple VLMs across three tasks, supported by human references and consistency analyses.

Critical weaknesses
  1. Limited novelty over existing benchmarks. IVQA-LD (ICML 2026) already evaluates VLMs on individuals with limb deficiencies, including visual grounding and limb-related understanding. The paper does not clearly establish how its tasks and findings provide substantial new insights beyond this closely related work.

  2. Missing negative samples in the Presence task. As acknowledged in Appendix F, the benchmark contains no examples of the "neither" category. This prevents evaluating whether VLMs falsely identify residual limbs or prosthetic devices when neither is present, limiting the completeness of the evaluation.

  3. Questionable validity of the Compatibility task. Compatibility is determined by predefined anatomical and device-category rules rather than actual device suitability. Furthermore, exact-set accuracy is sensitive to the number of valid candidates, making it difficult to distinguish reasoning failures from evaluation artifacts.

Questions for authors
  1. Novelty: How does this benchmark differ from IVQA-LD (ICML 2026) in terms of evaluated capabilities and empirical findings? What new insights does it provide beyond existing benchmarks?

  2. Presence evaluation: Why are examples from the "neither" category excluded? How does this omission affect the benchmark's ability to evaluate false-positive predictions?

  3. Compatibility evaluation: How are the compatibility rules validated against real-world device suitability? Can the authors analyze whether performance is affected by the number of valid candidates and candidate-set composition?

3 comments
AuthorsOpenReview ↗

Response to Reviewer enW2

Thank you for recognizing the importance of the research problem and the value of the evaluation.

1. Novelty concerns the capabilities made measurable and the resulting findings.

IVQA-LD evaluates residual-limb grounding and broader question-answering tasks, and studies model adaptation. Our contribution is a test-only diagnostic benchmark with new residual-limb and prosthetic-device masks, paired attribution formats, complete compatible-set recovery, and correctness-aware paraphrase analysis (§§3.3–3.5).

These measurements yield specific findings beyond a general observation that models struggle with disability-related images. On identical 500-item inputs, Qwen3-VL-4B-Thinking changes from 17.22% to 43.1% Attribution mIoU when the output format changes (§4.1; Table 4(b)). GPT-5 achieves 57.6% Partial but only 2.9% Exact in option matching, with 18.9% recall identifying substantial omission of valid matches (§4.2; Table 2). Qwen3-VL-235B repeats the same incorrect attribution throughout 47.28% of paraphrase groups (§4.3; Table 6(e)). These findings distinguish format sensitivity, incomplete set recovery, and stable errors. A shared target population does not make these evaluation protocols or findings equivalent.

2. The missing “neither” class is an explicitly stated coverage limitation.

Appendix B.2 describes the selection of images with sufficient visible limb evidence or body–device configuration information. Appendix B.5, Table 6(b), and Appendix F explicitly disclose that no targets belong to “neither.” The response vocabulary has four labels, while the evaluated targets support three.

Accordingly, the benchmark does not measure correct rejection when both evidence types are absent. It can still measure false attribution of an individual evidence type—for example, predicting residual-limb evidence in a prosthetic-only image. The missing negative class limits that specific evaluation scope; it does not invalidate the measurements on the supported classes.

3. Compatibility has a defined visual target, and candidate-count effects are not ignored.

Section 3.4 and Appendix B.6 define matching between annotated body-part and device categories. Appendix D.4 explicitly excludes personalized device recommendations and clinical suitability. Requiring clinical suitability validation would evaluate a different target from the one defined in this paper.

Table 7 already separates single- and multiple-category cases, predicted and target set sizes, and missed versus extra selections. Appendix C.9 reports Exact separately for nonempty and empty targets: 1.95% and 9.56%, respectively, for the 235B model. Poor recovery therefore persists within both groups. These are descriptive diagnostics; they do not establish invariance to target cardinality or candidate composition, and the paper does not claim otherwise.

Thanks for the response. I still have some concerns. Without the neither category, the Presence task cannot evaluate false positives when neither type of evidence is present. For Compatibility, it remains unclear how target set size and candidate composition affect exact-set accuracy. I keep my original rating.

AuthorsOpenReview ↗

Follow-up to Reviewer enW2

Thank you for specifying the two concerns behind your unchanged rating.

Presence. We agree that estimating how often a model reports limb evidence when neither type is visible requires images with a “neither” target. Appendix B.5 explicitly identifies this limitation. The existing evaluation answers a related question directly: how often does a model report an evidence type that is absent when the other type is present? From the confusion counts in Table 6(b), Qwen3-VL-235B incorrectly reported residual-limb evidence on 2,017 of 17,373 prosthetic-only prompts (11.61%) and prosthetic-device evidence on 1,692 of 7,302 residual-only prompts (23.17%). The missing class limits the estimate for images with neither type visible. The two error rates above remain measurable on the evaluated images.

Compatibility. Table 7(c) lets us say more about target-set size than the overall Exact score suggests. On 4,916 grids, the 235B model selected a nonempty proper subset of the correct options. Every one of those grids therefore had at least two correct options. Across the entire evaluation, the model recovered only 210 nonempty target sets exactly. Even if all 210 successes occurred on grids with at least two correct options, exact recovery on those grids could be no higher than 210/(4,916 + 210) = 4.10%.

Table 7(b) also shows that the model returned the correct number of options on 11.4% of grids but the correct set on only 3.07%. Even when it selected the right number of options, it usually selected the wrong ones. A controlled change in distractor composition would help explain which grids are harder. The published counts already establish poor recovery on grids with multiple correct options.

Given these results, could you identify which conclusion about the evaluated images or grids you consider unsupported, and explain how that concern leads to your recommendation of weak rejection?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.