- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper introduces a verifiable benchmark to measure whether AI models are good at handling uncertainty in tasks. However, it is extremely unclear and has questionable soundness since the benchmark construction and validity is not clearly explained. Because of this, I give a clear rejection.
Critical strengths
- Interesting Idea. It is interesting to attempt to evaluate models' abilities in resolving uncertain scenarios in a deterministic way. I believe this is much needed given the literature on ambiguity.
Critical weaknesses
- Benchmark construction is unsound. Since the paper proposes a benchmark for understanding how models act under uncertainty, it would be incredibly helpful if the authors spent more time convincing me that their benchmark construction is correct. Instead, I am given, if interpreted generously, the benchmark construction in less than 1 page which essentially just says "we constructed the benchmark." I cannot tell how accurate your benchmark is (e.g., whether it exhibits task, construct, outcome validity if you do not discuss its quality and quality control). Rather, because the entire paper is unclear and seems AI-generated (e.g., the entire paper refers to the benchmark as "V3", there are random requirements never introduced that are referenced like "the first three also cover H2 and H5" and I have no idea what H2 and H5 are since they're given a very vague reference in Section 2.2, a lot of jargon is introduced, lots of colons, very information dense but also glib writing, etc. What does "descriptive independence reference of 85.28%" mean? When was the reference introduced?). Based on this, I am left to believe that the benchmark and paper was one-shotted, which makes me doubt its soundness. It would be extremely helpful to spend more time discussing how the benchmark was created, how you ensured data quality, and what an example concretely looks like in the main text. As a result, all of your experiments are hard to interpret. In fact, even the appendix is not clear on how the benchmark was constructed.
- Lack of Uncertainty Quantification: The paper's main result figures (Figures 2 and 3) rely on differentiating between very small percentage point differences. It is hard to know whether the differences exhibited between models are statistically significant and whether the difference between preference, objective, and constraint problems are significant. Since the paper is a benchmark paper, it is incredibly important to quantify uncertainty.
Questions for authors
- Can you please elaborate on how your benchmark was constructed? Please be very clear about each step and justify validity and quality (see critical weaknesses).
- Can you include at least 1 qualitative example of a benchmark task, the models' actions in that task, and how it is scored?
Thanks for the clarification. I will maintain my original score.