For validating the grader (Appendix B), the authors asked experts to assess the judge’s assigned scores. Why not instead have experts assign their own scores independently and then report the judge’s accuracy against them? That is a far more standard way to validate LLM-judges and more trustworthy.
Furthermore, the expert acceptance rate is fairly low at 82%. What implications does this have for the benchmark? I didn’t see a sensitivity analysis of how robust benchmark conclusions are given this level of grader-accuracy.
It would be good to see a robustness analysis against other ways to combine the D / Q / R subscores. There seems to be no validation that your way of aggregating the scores correlates well with expert judgements of agent outputs.
The distinction between answer-quality subscore and reliability subscore should be better explained. What does a researcher conclude if their agent scores high on one but not the other?
How was the quality of the rubrics and reference answers evaluated/ensured? This paper does not convincingly establish the quality of its ground truth.
How does your grader check for the presence of hallucinated information in agent outputs, when this information is in statements that are not directly assessed by any rubric criteria?
The paper results are not reproducible due to the use of a proprietary harness. Why not also report some results in a standard open harness for comparison?
The reference-evidence refresh is an interesting idea, not a weakness, I am just concerned how you will ensure daily refreshes happen correctly. Could you describe the process that will ensure the refreshes run as specified? Somebody running this benchmark could easily get invalid results if it happens to not be properly refreshed at that time.
Fig 3 is not very informative, because the model IDs are hard to read and missing from B & C. Would use more types of markers and a legend to indicate specific models/families.
Dear Reviewer cxHX,
Thank you for pointing this out. We did disclose our AI use in the OpenReview submission form, but inadvertently omitted the corresponding statement from the main text. We will add the AI use statement to the paper in the revised version.
We would also welcome any further specific comments on the paper’s content, and are glad to address them during the discussion.
Best regards, Authors