- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 3
Read review
Paper summary
The paper proposes and studies the idea of 'evidential entitlement', which is a bit more general than citation (which sources that were inputs to the LLM, not sources that were cited in the final output). The authors variously discuss ideas around citation and evidence, with some theory and targeted interventions, 'entitlement flipping'
Critical strengths
In general, the study of citations and evidence is interesting and useful.
Questions of coverage and scope of the retriever, and how that interacts with a lack of evidence, are an important problem in human-LM interfaces.
Critical weaknesses
I suspect most, or all, of this paper is AI-generated. Worse, I suspect that the ideation and execution of the paper are AI-generated. I've seen such papers enough to have a general sense of this - the paper includes spurious theory despite not really being a theory paper, detailed experiments that are based on niche ablation-type questions, and so on. AI generation may not be disqualifying in and of itself, but AI-generated papers that are unpleasant for reviewers to read are a disqualifying criterion.
I also don't really find the particular question they ask in this paper to be interesting. Evidence is, of course, interesting, but why focus on what evidence was given to the LM generator? Having read the paper, I've yet to understand why this is a key concern in generation reliability..
I also find the experiment design to be suspect. For a question that's tied to human downstream decisions, the eval designs are strange - no human eval or user studies, no downstream decision evals..
Questions for authors
What is the purpose of theorems 1, 3, 4. Can they not be entirely removed without affecting the main claims of the paper?
Do you have evidence that evidential entitlement is useful for downstream user decision making, users making calibrated decisions, or other more grounded outcomes?
Evidence Delivery, Theoretical Results, and Downstream Decisions
Thank you for your feedback on the paper’s motivation, presentation, and evaluation.
W1. Concerns about AI involvement and readability
The submission form discloses AI assistance with writing, literature discovery, research ideation or execution, and mathematical proofs. Page 10 describes language and structure editing, literature discovery, analysis and figure code editing, and feedback on presenting experiments. As disclosed, the authors reviewed the assisted text and code, checked citations against their sources, and verified the numbers and figures against the original outputs. The authors take responsibility for the manuscript and its conclusions.
W2. Why focus on what evidence reaches the generator?
The delivery record tells users where to look next. In archived case 003, the assistant said it could not confirm whether the Kubernetes v1.30.0 API specification defines
AppArmorProfile. The first delivered passage explicitly lists that definition, so the user can check the answer against the source. In the HotpotQA example, the answer and citation are correct, but the passage connecting the question to the answer was never delivered. The first case calls for reviewing or correcting the answer. The second calls for retrieving the missing evidence. We will introduce these examples before the definitions and experiments.W3. Does the evaluation match the paper’s motivation?
The offline replay in Q2 evaluates whether the record leads an automated workflow to select the appropriate next action. Measuring the effect on users’ decisions requires observing how they use the record.
Q1. What would be lost by removing Theorems 1, 3, and 4?
Removing the theorems would leave the observed collision counts intact and remove three steps linking those observations to the design of our records.
Q2. What evidence is there that the record improves downstream decisions?
During a Kubernetes upgrade review, an engineer who sees “unable to confirm” must decide whether to close the check for AppArmorProfile or send it back for another search. In case 003, the first delivered passage lists the exact definition, so the engineer can close that specific check and move on to other compatibility questions. If the record shows a timeout, the check stays open and the engineer can search the requested version again. The same answer can lead to either action. We replayed four possible actions on 1,728 archived executions. The answer-only rule matched the predefined action in 666 cases, validated citations in 760, and the full execution record in 1,728. These results measure whether the system can direct the next step. The effect on time spent and repeated searches requires measurement in actual use.