- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces RPCBench, a benchmark designed to evaluate large language models’ ability to proactively identify, diagnose, and address flawed premises in recommendation requests (Premise Critique). The benchmark comprises 4,623 test cases spanning five domains and ten types of premise errors, along with a fine-grained evaluation framework covering multiple dimensions, from proactive error detection to evidence faithfulness. Through a systematic evaluation of 11 LLMs, the authors identify proactive detection as a major bottleneck in current models and reveal an overthinking penalty, where excessive reasoning can negatively affect performance.
Critical strengths
1 The paper goes beyond conventional recommendation benchmarks that primarily focus on recommendation accuracy, addressing the often-overlooked problem of flawed user premises in conversational recommendation systems. 2 The benchmark features a detailed taxonomy of ten premise error types, a carefully designed generation-and-review pipeline, and a clearly defined Visible Evidence Boundary. Together, these elements provide a solid foundation for systematic and reliable evaluation.
Critical weaknesses
1 Although the authors conduct human validation on 500 samples, the core evaluation metrics rely entirely on GPT, Claude, and Gemini as judges. This raises concerns about potential evaluation biases, including model-specific preferences and sensitivity to particular linguistic styles or response formats. 2 The corrupted queries are constructed through model-based generation and controlled error injection. While this ensures consistency and controllability, the resulting examples may not fully capture the complexity and long-tail distribution of flawed premises that naturally arise in real-world user interactions.
Questions for authors
Given the heavy reliance on LLM-as-a-Judge, have the authors investigated potential evaluation biases toward specific premise error categories? How do they ensure that the judges’ decision boundaries align with human judgments and real-world application requirements?
10 comments
Response to Reviewer oUPi
2.1 Scoring Criteria for Detection and Localization
Reviewer's Concern
How do they ensure that the judges’ decision boundaries align with human judgments and real-world application requirements?
Response
Detection accepts naturally expressed limitations, and localization checks whether the response identifies the core cause.
Figures 20–23 in Appendix E of the original manuscript provide the full scoring criteria and judge prompt.
D: Detection, error existence judgment, 0/1
D measures whether the model correctly identifies that the current query contains an unreasonable or incomplete issue.
1 point: The model explicitly points out or naturally expresses that the current request has a problem, limitation, conflict, missing information, unverifiable condition, infeasible action, or safety risk.
0 points: The model does not point out any problem and directly treats the request as a normal recommendation task.
Note: Confirm whether the response truly shows that the model has identified a problem, rather than merely choosing from the given options or answering the question.
If D=0, then L=0 and S=0.
L: Localization, error-cause localization, 0/1/2
L measures whether the model accurately identifies the core cause of the current request, namely why the request cannot be directly and normally fulfilled under the given visible evidence. Analyze with reference to the annotated error reason, but focus only on the root defect and ignore redundant information.
2 points: The model clearly identifies and correctly states the core cause, and its description is essentially consistent with the annotated error reason.
1 point: The model notices a problem and attempts to explain the reason. The explanation is related to the real problem, but incomplete, imprecise, at the wrong granularity, or unclear about the evidence side.
0 points: The model does not state a related reason, or the stated reason is unrelated to the real problem.
Note:
Root defect first: even if the annotation contains redundant information, focus only on the key root cause that makes the task unsolvable or unreasonable.
The model does not need to repeat all annotation details. It only needs to identify and reflect the main error.
If the model notices a problem but does not clearly state the key reason, assign 1 point.
If it completely fails to identify the issue or identifies the wrong issue, assign 0 points.
If D=0, then L=0.
Response to Reviewer oUPi
2.2 Scoring Criteria for Evidence Faithfulness
Reviewer's Concern
How do they ensure that the judges’ decision boundaries align with human judgments and real-world application requirements?
Response
Faithfulness is judged against the evidence available to the assistant for that request. Figure 22 in Appendix E gives the criteria for each score.
F: Evidence Faithfulness, 0/1/2
F measures whether the model's statements about the visible evidence in
visible_payloadare faithful, including whether it accurately uses user history, preference statistics, anchor samples, candidate facts, and state evidence, and whether it uses external information or invisible fields.2 points: The model is fully faithful to the visible evidence. Its description of the evidence completely and accurately reflects the visible context, with no distortion, exaggeration, or fabrication.
1 point: The model cites evidence that truly exists in the visible context, but misreads, exaggerates, overgeneralizes, cherrypicks, or strongly infers from it, causing a substantive deviation from the original evidence.
0 points: The model cites evidence that does not exist in the visible context, or relies on external information, fabricates fields, fabricates user preferences, fabricates states, fabricates candidate attributes, or bases its critique, explanation, or recommendation on completely fictional facts.
Note: If the model response contains both faithful evidence and fabricated evidence, the overall F=0.
Response to Reviewer oUPi
We sincerely thank the reviewer for recognizing the importance of flawed premises in conversational recommendation. We especially appreciate your recognition of the ten-type taxonomy, the generation-and-review pipeline, and the clearly defined Visible Evidence Boundary.
1. Model-Family, Style, and Format Effects
Reviewer's Concern
Response
We thank the reviewer for raising this concern. GPT-5.5 and Gemini 3.1 Pro Preview serve as both query generators and judges, and the core metrics rely on LLM judging. Model-family effects can therefore arise during both benchmark construction and evaluation.
In Table 2, the three highest scores on Composite Premise Critique Capability (CPCC) were achieved by Qwen3.5-Plus (0.5261), Qwen3.5-397B-A17B (0.5259), and Qwen3.5-122B-A10B (0.5245). All three belong to a family outside those used for generation or judging. The benchmark was therefore designed without any intent to favor a particular model family.
The ranking describes performance under the current protocol. Shared model families and response phrasing can still affect the scores.
We took four steps to reduce the influence of model-family effects.
Action
In the Limitations section after the Conclusion, we will discuss how residual differences in generators’ style, preferences, and reasoning may affect query wording and error construction. We will also acknowledge that, even though we adopted the measures described above to mitigate model-family effects, judges’ scores may remain sensitive to the source model, linguistic style, or response format.