- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The authors introduce SceneActBench, a benchmark for testing the ability of foundation models to reconstruct objects in 3D scenes from single inputs. The authors evaluate their benchmark on a suite of foundation models.
Critical strengths
The evaluation is thorough (both in main paper and in the Supplementary).
Critical weaknesses
Novelty with respect to related works: The paper is entirely missing discussion of how the benchmark compares to related works. The only mention of related works is three small paragraphs in the Appendix and Table 1 in the main paper. This table, features simple checks and crosses, and there is no mention of the table in the text. As the paper is currently presented, I cannot infer the novelty or utility of this benchmark with respect to related works. If this benchmark is simply a unification of skills that are previously tested in other benchmarks - then it should be stated as such (though the questions of utility remain as these are tasks that are well-covered in other single-task benchmarks).
Lack of Analysis/qualitative examples: As I understand it, the goal of the benchmark is to probe the ability of foundation models to model 3D objects based on visual inputs. If this is the case, then the paper desperately needs some qualitative examples with corresponding analysis. Currently, the analyses are purely restricted to comparing values from a table (and a few isolated qualitative examples in the appendix) - but this gives the community very little insight to the current issues these models encounter. A more thorough analysis, with many qualitative examples, would be far more valuable than many of the details that are currently in the main paper (e.g. the file types of the benchmark).
Significant AI-generated content: The text throughout the paper reads in a very AI-generated manner. The figures in the whole paper have the very typical "AI-generated" style. The textual details in the paper are typical of AI-writing - there is an emphasis on file types and formatting, as opposed to the scientific and technical questions the dataset is attempting to address. The current presentation makes the paper and subsequent benchmark hard to appreciate.
Questions for authors
Please see weaknesses.