- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This benchmark attempts to measure how often AI make decisions when the evidence or permission given is not sufficient for making those decisions. However, the benchmark is unmotivated by the literature, very unclear, poorly constructed, and poorly analyzed so I mark clear reject.
Critical strengths
None
Critical weaknesses
- Ungrounded Task and Scenario Construction: The benchmark relies on the EACTRN axes to establish ground truth, but it's not clear to me why these axes are the ones to focus on. Only A (operational authority) is grounded in the literature. Without literature grounding, ECTRN seem like random axes generated by AI to try to be comprehensive. Further, it's not explained how the tasks in the benchmark are generated or whether they're necessary at all or relate to downstream impacts. This makes me believe this benchmark is a benchmark for creating a benchmark's sake rather than attempting to measure something useful. Human study is very unclear and there is no appendix explaining additional details.
- Shallow Analysis: This is a benchmark paper and therefore it should provide thorough analysis of model performance on the benchmark. Instead, the results section just basically tells me that models over-authorize. The error analysis is just 2 confusion matrices that don't actually dig into the reasoning traces of these models. Much deeper review is needed. Please also quantify uncertainty when possible.
- Very Old Models: The paper only evaluates Llama 3.3 70B which released in Dec 2024 and Mistral-Large-Instruct-2411 which released in Nov 2024. Since then, frontier models have gotten much better. I cannot tell if this benchmark would be saturated by current models (if it's saturated, it's not a useful benchmark).
Questions for authors
- In organizational settings, is liability or other accountability actually defined by EACTRN axes? If not, why focus on EACTRN here?
- Why did the models over-authorize? I would like a much deeper analysis than currently provided.