- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3(+1)
Read review
Paper summary
The paper introduces UNICORNS, a benchmark for evaluating whether 3D-LLMs respond appropriately to ambiguity, unavailable information, and false premises without making unsupported claims. It also proposes HORN, a post-training approach combining task and obligation rewards with supervised control replay. Experiments evaluate existing models and examine improvements through ablations, controlled scene edits, and additional generalization tests.
Critical strengths
-
The motivation is convincing. Reliable 3D question answering requires recognizing what the available observations can and cannot support. Evaluating clarification, uncertainty, and premise correction alongside ordinary answering addresses a meaningful gap in conventional benchmarks.
-
The work is fairly complete, with paired answerable controls, scene-disjoint splits, training ablations, and controlled scene-edit experiments. In particular, the scene edits provide useful evidence that model responses follow changes in scene content. HORN also improves challenge performance while preserving answerable-control accuracy on the main backbone.
Critical weaknesses
-
The validation of the underlying evidence needs more detail. Although the paper states that retained samples undergo human review, it is unclear whether reviewers independently inspect the original scenes to verify both the correctness and completeness of the structured evidence. This matters because the veto rule could reject a valid scene detail simply because it is missing from the evidence supplied to the judge. Agreement between human and LLM judges does not resolve this concern if both rely on the same evidence package.
-
The paper does not report human performance on the benchmark itself. The 480-response audit evaluates human–judge agreement on model outputs, rather than humans answering the questions. A human reference under comparable observation conditions would help establish whether the questions and response requirements are reasonable and contextualize the remaining model errors. For a benchmark focused on evidence sufficiency, this is an important missing validation.
Questions for authors
-
Could you clarify the human review procedure for dataset construction, including whether reviewers inspect the original scenes, how evidence omissions are identified, and how disagreements are resolved? Did the separate 480-response audit also allow access to the original scenes?
-
Could you provide human answering performance on a representative subset, with access to comparable scene observations but without the ground-truth evidence or reference answers? An analysis of human failures would also help distinguish model limitations from ambiguity in the questions or scoring criteria.
Minor Suggestions for Improving the Revised Manuscript
Thank you for the revised manuscript. The additional experiments and analyses have improved the paper. I have a few minor suggestions that could further strengthen its clarity and reproducibility.
These are minor suggestions intended to improve the presentation and reproducibility of the work. I am inclined to accept this paper.