- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper focuses on a new field, the images of individuals with limb deficiencies. The problem is interesting and reasonable. Based on the problem, the paper introduces InclusiveVLM-LimbEvidence, a benchmark for evaluating vision-language models on images of individuals with limb loss in real-world settings. The authors incorporate new annotations for residual limbs and prosthetics, structuring the evaluation around presence detection, body part attribution, and matching with candidate device categories, while assessing several mainstream VLMs. Results indicate that coarse-grained recognition capabilities do not guarantee accurate identification of body parts and laterality, nor do they ensure the correct retrieval of the full set of matching options. Furthermore, consistent responses to rephrased queries may still be incorrect.
Critical strengths
- The problem setting is interesting and useful.
- Separating evidence detection, spatial attribution, and structured matching helps failures analysis, which is good for diagnostic evaluation.
- The new annotations extend the source dataset meaningfully and the experiments reveal some useful preliminary results.
- The failure analysis for current SOTA VLM is beneficial.
Critical weaknesses
- In Appendix C9, the experiment results show that constant predictions achieve 65.79% Presence accuracy, predicting prosthetic-only, and always predicting an empty set yields 14.67% Diversity Exact. This should be compared with primary results and more analysis is needed to distinguish the true performance from label priors.
- Errors from visual perception have not yet been fully distinguished from those caused by prediction formatting or task interpretation. On the same set of 500 samples, the Attribution mIoU for Qwen3-VL-4B-Thinking improved from approximately 17.2% to 43.1% when using a free-text format. This significant difference indicates that the label expression affects the performance substantially.
Questions for authors
- How do model results compare with simple baselines across all primary metrics, particularly when empty and nonempty targets are evaluated separately?
- How much attribution error remains under a common, natural-language output interface?
- Can the authors provide independent annotation-agreement statistics and executed option-permutation controls?
4 comments
Thanks for clarifying the FreeForm results. The comparison on the same 500 examples answers my question about how much attribution error remains when models can respond in their own words.
My main concern is still how much the models gain from seeing the image. Always predicting “prosthetic only” already gives 65.79% on Presence. I’d find the results easier to interpret with simple baselines evaluated on the same examples as each model, and with a comparison of model performance with and without the image. I’d also like to know how often annotators agreed before adjudication and whether models change their answers when the options are reordered. I keep my original rating.
Follow-up to Reviewer 27Zk
Thank you for confirming that the comparison on the same 500 examples resolves your question about attribution under natural-language output.
Presence baselines on the prompts actually scored. For Qwen3-VL-235B, the comparison is exact because all 26,406 responses were valid. The model answered 19,601 prompts correctly, while always predicting “prosthetic only” answers 17,373 correctly. Table 6(b) shows where the difference comes from. The model correctly answered 4,388 prompts from the two minority classes and gave up 2,160 correct answers on the majority class, for a net gain of 2,228 correct answers, or 8.44 percentage points, on identical prompts.
Qwen3-VL-30B-A3B-Instruct achieved a parsing success rate above 90% across the 26,406 Presence prompts. Its accuracy on successfully parsed responses was 23.9%. Even in the worst case for the majority baseline, if every parsing failure occurred on a “prosthetic only” prompt, always predicting “prosthetic only” would still achieve at least 61.99% accuracy on the exact prompts scored for the 30B model (Table 6(b)). Thus, even under the assumption most favorable to the model, it falls more than 38 percentage points below the majority baseline.
Image use, annotation agreement, and option order. Quantifying the gain from seeing the image requires running the same model on the same prompts with and without the image. We have not reported that experiment. Appendix B.4 documents second-annotator review and adjudication, but the submission gives no agreement rate before adjudication. Appendix B.6 documents randomized option order during grid construction, but the same grids were not evaluated under repeated permutations, so we cannot report how often reordering changes a prediction.
Could you clarify whether you dispute the observed body-region and option-selection errors, or whether your recommendation turns on the further question of what causes those errors? The former can be checked against the recorded predictions and targets. The image-removal and permutation experiments would address the latter.
Thanks for clarifying. I agree that the 235B model beats the majority baseline on Presence and gets the evidence type right more often than the body region. I also accept the body-region and option-selection errors you report. My concern is how much seeing the image helps the model identify the evidence in the first place. Since always answering “prosthetic only” already gives 65.79% accuracy, I’d like to see how the same model does on the same prompts without the image before drawing the broader conclusion that models “often detect limb evidence.”
Response to Reviewer 27Zk
Thank you for recognizing the benchmark’s value and the usefulness of its diagnostic decomposition.
1. The baseline comparisons reveal limitations in current models; they are not omitted from the interpretation.
Appendix C.9 reports the 65.79% Presence majority baseline and 14.67% always-empty Diversity Exact baseline. All models use the same task-specific test inputs, so these references apply to their shared pools. Only GPT-5 (77.2%) and Qwen3-VL-235B (74.23%) exceed the Presence majority baseline, and all model Exact scores fall below the always-empty reference. We do not treat the remaining scores as evidence of above-prior recognition or set recovery.
The requested empty/nonempty distinction is already partly available. Table 7(c) and Appendix C.9 give the 235B model’s Exact as 210/10,765 = 1.95% on nonempty targets and 177/1,851 = 9.56% on empty targets. Poor recovery therefore occurs within both groups, rather than resulting solely from their mixture. A complete baseline panel across all primary metrics is not reported.
2. Substantial attribution mismatch remains under natural-language output.
Section 4.1, Table 4(b), and Appendix C.5 directly answer this question. On the same 500 image–target pairs, Constrained→FreeForm mIoU is 17.22→43.1 for Qwen3-VL-4B-Thinking, 15.87→44.7 for Qwen3-VL-8B-Thinking, and 26.51→30.5 for GPT-5. Parsing succeeds for 467/500, 452/500, and 417/500 responses, respectively.
These results establish a format effect, while also showing that natural-language output does not resolve attribution. They do not establish that every remaining mismatch arises exclusively from visual perception. Appendix D.2 explicitly distinguishes observed task errors from such a causal interpretation.
3. Annotation review and construction-time randomization are documented procedures, not unreported experiments.
Section 3.3 and Appendix B.4 describe second-annotator review, re-annotation, and classifier-supported adjudication. Appendix B.6 documents randomized option order during construction.
Independent annotation-agreement coefficients and executed repeated-permutation results are not reported. We do not present adjudication as an independent agreement measurement, or construction-time randomization as demonstrated permutation robustness. The existing evidence supports reviewed targets and the reported model behavior under the fixed evaluation protocol.