acceptodds
Under review as a conference paper at ICLR 2027

Do LLMs Consider Document Credibility When Relying on Retrieved Evidence?

Abstract

Evaluations of retrieval-augmented generation (RAG) typically examine whether large language models extract retrieved evidence, resolve conflicts, and detect when evidence is insufficient. These settings assume that the retrieved document is trustworthy, while prior work on credibility relies on external cues such as source metadata. We ask instead: when a retrieved document provides evidence whose veracity the model cannot determine, but contains credibility flaws elsewhere, does the model reduce its reliance on that evidence? We call this capability credibility-aware evidence reliance and introduce CREDO (Credibility-aware Reliance Evaluation on Documents) to measure it. CREDO pairs real retrieval-dependent questions with documents that retain the target evidence but carry five types of intrinsic credibility flaws. The truth of the target evidence is unknowable to the model by construction, so document credibility is the only rational basis for reliance. Measured with four response-level behavioral metrics, most open-weight and proprietary models adopt the flawed document's claim in 85% to 99% of responses and identify credibility flaws in fewer than 10%, and more capable models are not more prudent. Furthermore, frequent search calls co-occur with low flaw identification, pointing to habitual tool use rather than credibility assessment. Prompting yields limited gains, whereas fine-tuning on synthetic evidence-reflection trajectories improves credibility-aware evidence reliance and performance on general retrieval-based QA tasks.

Primary area:
datasets and benchmarks
Keywords:
Credibility-aware Evidence Reliance, Retrieval-Augmented Generation, LLM Evaluation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_LzweOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper investigates whether LLMs reduce their reliance on a document's target claim when the document contains intrinsic credibility flaws. It introduces CREDO, with 1,292 retrieval-dependent yes/no questions and 7,752 document variants spanning opposing claim directions and one to three flaws. A controlled search tool returns fixed, inconclusive background material. Across 22 models, the study measures claim adoption (ADOPT), flaw identification (IDENTIFY), search (SEARCH), and overclaiming (OVERCLAIM). It reports that most models readily adopt the supplied claim, that supplying the same document as a tool response increases adoption, and that evidence-reflection SFT improves these metrics.

Critical strengths
  1. Separating whether a model adopts a claim, identifies a flaw, or expresses uncertainty is a useful idea.

  2. The comparison between supplying the same document in a user message and in a tool response, and the observation that Llama models search by restating the question, are of some interest.

Critical weaknesses
  1. The core construct is self-contradictory, and the paper's claims do not hold under either reading.

The paper gives two incompatible descriptions of where the flaws are placed:

  • Main text (Abstract; Section 1, p. 2; Section 2.2, p. 3; Section 5, p. 9): documents "retain the target evidence", flaws are located "outside the evidence relevant to the target claim", and none "directly contradict" that claim.
  • Appendix A.7.3 (pp. 23–24): the synthesis prompt requires a document "whose critical support chain contains one subtle gap"; the generator must "choose one conclusion-critical bridge", "put the assigned and only gap in that bridge", and copy the cue span "from the weak premise before the final claim". The review prompt (p. 24) further requires that, once the cue-bearing material is removed, the remaining body must not independently entail the conclusion.

The appendix's own examples match the latter description. In Figure 9 (p. 21), the inconsistent Slovenian yields and the conflicting fulfillment totals sit directly on the data needed to answer the question. In Figure 1, the conflicting trial-operation dates also bear directly on the opening date.

Reading A (main text): the flaws do not affect the target claim. In this case ADOPT↓ lacks a normative basis. For a yes/no question with no other evidence, a source that is not fully reliable, but not deliberately misleading, still raises the probability that its claim is true, so answering in the document's direction remains the best single guess. The paper itself states that the flaws "warrant caution in relying on its target evidence, even if the target claim is true" (p. 1); caution calls for expressing reservations, not for changing the answer. Because ADOPT also counts abstention as non-adoption, ADOPT↓ effectively rewards abstention. Gemma's adoption dropping to 0.2–0.7% after SFT (Table 14) illustrates this problem. In addition, the paper lacks an ablation with zero-flaw or repaired-flaw documents that retain the same target evidence, so it cannot distinguish a model that is sensitive to flaws from one that distrusts everything. Under this reading, neither the value of the dataset nor the meaning of the main metric is established.

Reading B (appendix): the flaws lie in the support chain of the conclusion. In this case the document genuinely fails to support its conclusion, non-adoption is the correct behavior, and ADOPT becomes meaningful. However:

  • The task then reduces to detecting whether the context suffices to support the answer, i.e., the sufficiency dimension that the paper explicitly claims to go beyond (Sections 1 and 5). It overlaps substantially with RGB, FaithEval (in Related Work Section 5), and Sufficient Context (Joren et al., ICLR 2025), and the claimed novelty no longer holds.
  • The title, abstract, introduction, Figures 1 and 2, and the positioning in the related-work section do not describe the actual data.

Therefore, under Reading A the dataset has little value and the main metric is normatively unfounded; under Reading B the novelty claim fails and the paper misdescribes its own data. This cannot be explained as an outdated template or unclear documentation. Whichever reading the authors confirm, the paper requires a substantial rewrite.

  1. The metric definitions cannot be reconstructed and do not align with the validated fields.
  • In Section 3.2, IDENTIFY only requires a specific description of a flaw, whereas the judge additionally requires explaining how the flaw weakens support for the conclusion (p. 25). This merges recognition with assessing its implications, which the paper sets out to distinguish.
  • The formal OVERCLAIM metric requires an uncertainty variable U, and per-type recall requires a matched flaw set F, but neither field appears in the judge's output schema. Appendix A.7.4 explicitly states that the avoided_overclaim field differs from the main-text OVERCLAIM metric, yet gives no mapping between them.
  • Human validation covers the raw judge fields, not the final metrics. Expert-consensus agreement is lowest for avoided_overclaim (85.23%). The valid_evidential_inference field is validated but not used by any metric.
  1. The construction and quality-control specifications directly conflict with the main text in several places.
  • Number of flaws. The review prompt states that "each document has one primary flaw" and includes a MULTIPLE_DEFECTS rejection code (pp. 24–25), whereas the main text states that each document receives three flaws of distinct types (Section 2.2). Under the appendix's review criteria, the three-flaw documents described in the main text should all have been rejected.
  • Undisclosed data language. The synthesis prompt requires documents of "650–1,050 Chinese characters" (p. 23), and all appendix prompts are presented as English translations, indicating that the data are in Chinese. The main text, Figure 1, and all examples are presented in English, and this is never stated. Were the target models evaluated in Chinese or in English?
  • Inconsistent query-selection criteria. The main text states that long-tail or time-sensitive queries are retained, but acceptance requires scores of at least 4 on long-tail-ness, temporal variability, and specificity alike (Section 2.2; Appendix A.7.2).
  • Source of the target evidence. The synthesis prompt allows synthetic records to carry the desired conclusion, attributed to neutral repositories or project logs (p. 23); that is, the support for the target claim is itself fabricated. Figure 2 and Section 2.2 emphasize that background material comes from real retrieval with no added factual content, which may give readers an impression of realism, while the synthetic nature of the target evidence is not stated.

Note on the scope of the response: Weakness 1 affects the paper's core conclusions, and I ask the authors to focus on it. For the items in Weaknesses 2 and 3, as well as other inconsistencies in the paper, the authors may respond selectively; a point-by-point reply is not required. These points are raised to help the authors clarify the actual experimental setup and strengthen the soundness of the paper.

  1. Lack of reproducibility.

Given the issues above, the only statement on availability is the sentence "Code and data will be released" at the end of the abstract; no anonymous code repository or supplementary material is provided. All appendix examples are marked as having altered "names, dates, values, and wording" (pp. 19–20), so the paper contains not a single unaltered benchmark instance, and the prompts are provided only as English translations. Under these conditions, none of the points above can be independently verified, which further weakens the soundness of the paper.

Questions for authors
  1. Does the actual data correspond to Reading A or Reading B? If A, please explain why flaws that do not affect the target claim should change the model's answer, i.e., the normative basis for ADOPT↓. If B, please explain how this work differs from evidence-sufficiency detection, and revise the title, abstract, and introduction accordingly.

  2. How are "each document has one primary flaw" and the MULTIPLE_DEFECTS rejection code in the review prompt reconciled with three flaws per document? In what language are the data constructed and the models evaluated?

  3. How are A, U, and F obtained from the actual judge outputs? Please provide the exact mapping and human agreement on the final metrics.

  4. Please report supported-answer, abstention, and false-flaw-identification rates for the base and SFT models on zero-flaw/repaired-flaw documents and on the conclusive-evidence control.

  5. The ICLR 2027 Author Guidelines allow the paper's content and supplementary materials to be revised during the discussion period. Please provide anonymized code, the complete data (or at least a representative subset), and unaltered original examples by updating the supplementary materials.

3 comments

Thank you for the detailed review. We address the central distinction and implementation questions below.

Q1. Credibility concerns do not logically refute the target claim

A flaw not logically refuting a claim does not mean it has no bearing on trust in the document. CREDO requires that flaws must not establish the opposite answer. Appendix A.7.3 explicitly states: "The gap may weaken the current claim but must not prove the opposite answer." Detecting a flaw therefore does not establish that the target claim is false or necessarily require reversal or abstention.

When a model cannot answer from parametric knowledge and receives an answer-bearing document of unknown provenance, its options extend beyond complete refusal and unconditional acceptance. It can attempt tool-based cross-verification. Even if the returned information remains inconclusive, it can identify contradictions or provenance concerns as leads for further investigation and communicate reservations about evidence quality.

Thus, the proposed implications for answer behavior under Readings A and B do not necessarily follow: a flaw that does not negate the claim need not be ignored, while weakened support does not make abstention the only appropriate response. We assess whether models identify credibility concerns and reflect them in their answers, not whether they automatically change answer direction.

Section 3.2 explicitly allows qualified adoption: adopting the claim with evidence-related uncertainty counts toward ADOPT but not OVERCLAIM. We will clarify flaw placement and evidential effects and avoid presenting lower ADOPT alone as improvement. We recognize that the non-refutation requirement alone does not resolve the separate question of where flaws lie relative to the support chain; the construction description must make that distinction explicit.

Q2. Flaw counts and evaluation language

Section 2.2 describes constructing three distinct flaws and removing complete flaw paragraphs to produce nested variants. The 1,292 questions have two claim directions and three flaw-count conditions, yielding 7,752 directional instances (Table 5). Table 4 distinguishes five top-level categories from finer construction labels.We have publicly released the relevant data for inspection.

Q3. Metric extraction and human agreement

Section 3.2 defines A as answer direction, F as identified annotated flaws, U as explicit evidence-related uncertainty, and S as observed search activity. The appendix stance labels can map to yes/no/abstain, and matched cue IDs in detection_evidence provide information for F.

We will release the evaluation and aggregation code to specify the exact sources and mapping of A, F, and U. In particular, detected_injected_cue requires identification and explanation of a flaw's implications; avoided_overclaim is an auxiliary confidence judgment whose complement cannot simply replace OVERCLAIM; valid_evidential_inference diagnoses understanding of evidential implications.

Table 7 reports expert-consensus agreement: Answer 100.00%, Flaw Identification 95.18%, Evidential inference 97.83%, Avoided-overclaim 85.23%, and overall 94.46%. These validate raw judge fields, not all final aggregated metrics.

Q4. Controls and post-SFT response behavior

Appendix A.2 and Figure 7 assess standard QA before and after SFT under a common retrieval setup; substring EM performance is preserved. We additionally sampled 100 questions from each of seven QA datasets, totaling 700, and manually reviewed responses from both models after evidence-reflection SFT:

SFT modelUncertainty while answeringComplete refusal
Qwen3-8B10.57%0%
Gemma-4-31B5.14%0%

These are aggregate sample statistics, not full-test-set or per-dataset estimates. Uncertainty while answering is distinct from complete refusal. This evidence concerns external QA; it does not supply the requested matched-control supported-answer and false-flaw-identification rates.

For zero-flaw documents, it is difficult to guarantee that collected examples are entirely correct and free of credibility flaws, especially for highly time-sensitive questions. Consider, for example, a hypothetical question about whether an unverified allegation concerning a public figure is true. A model may be unable to give a definite yes/no answer without refusing to engage with the question, it can discuss the available information and its limitations. Extensive online discussion does not itself establish the allegation’s truth. As LLMs are increasingly used as search engines, we believe that evaluating model behavior in such situations is valuable.

Q5. Anonymous resources

The questions, documents, and flaw annotations are available at: https://huggingface.co/datasets/anonymous-submission-01/CREDO/tree/main

We hope you are doing well. We sincerely appreciate the thoughtful feedback and constructive suggestions you have provided.

With the discussion deadline approaching, we would be grateful if you could review our responses when you have a chance and let us know whether any concerns remain. Your feedback would allow us sufficient time to address additional questions and provide any further information that would support the discussion.

If our rebuttal and manuscript revisions have resolved your concerns, we would sincerely appreciate your consideration of an increase in your rating.

Thank you again for your time and consideration. We look forward to continuing the discussion with you.

Thank you for the response. However, it seems the author has not adequately addressed my concerns regarding the internal contradictions in the core argument (W1, Q1). Consequently, I intend to maintain my current score, and I would also like to draw the Area Chair’s attention to the fact that the author has failed to resolve these issues.

Reviewer_LtiJOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The authors present a benchmark for credibility-aware post-retrieval answer generation in RAG. Specifically, it is aimed at assessing the LLMs on their ability to identify six types of logical and factual flaws in the retrieved documents with the answer and calibrate their response appropriately. The benchmark construction involves: (i) normalization of questions into yes/no answerable, (ii) selection of relevant background passages, (iii) consolidation of background passages with answer-containing passage into a coherent whole, and (iv) grafting one to three of the six types of flaws into the passage. The evaluation constitutes four major metrics: Claim Adoption Rate - answers adopting the document's claim, Flaw Identification Rate, Search Rate - instances with at least one search tool call, and Overclaiming Rate - yes/no answer without warning uncertainty.

Critical strengths
  1. The paper focuses on a very timely and important aspect of LLM reliability under uncertainty. Particularly, how LLMs express skepticism under the presence of inconsistencies in the source documents. This could be important for multi-agent communication where interaction between various agents could contain such apparent inconsistencies stemming from different sources of information.
Critical weaknesses
  1. While the work shows that SFT improves the various relevant metrics, it does not show if the performance is preserved when 0 flaws are present.
  2. The scope of the tool call is deliberately made limited to isolate credibility awareness. However, the ability of the LLMs to resolve these flaws is not explored.
  3. I am not quite convinced about the claim adoption rate metric. Despite inconsistencies in the documents that do not directly affect the veracity of the answer, why must the LLM abstain from answering? Granted, a warning should be raised by the LLMs about the exact nature of the flaws. I think this is quite central to the proposed evaluation framework.
Questions for authors
  1. I am not quite convinced about the claim adoption rate metric. Despite inconsistencies in the documents that do not directly affect the veracity of the answer, why must the LLM abstain from answering? Granted, a warning should be raised by the LLMs about the exact nature of the flaws and its relationship with the credibility. I think this is quite central to the proposed evaluation framework.
2 comments

Thank you for your thoughtful review. We clarify capability preservation, the controlled tool setting, and the interpretation of claim adoption below.

W1. Does SFT preserve performance without planted flaws?

Appendix A.2 and Figure 7 evaluate Qwen3-8B and Gemma-4-31B before and after evidence-reflection SFT on seven external QA benchmarks, using the same retrieval setup and Wikipedia index. Performance measured by substring EM is preserved. These experiments assess standard retrieval-based answering without CREDO's deliberately planted flaws.

We also sampled 100 questions from each of the seven datasets, totaling 700 questions, and manually reviewed both models' responses. These are the checkpoints after evidence-reflection SFT, not the base models. We distinguished expressions of uncertainty while answering from complete refusal.

Model after evidence-reflection SFTUncertainty while answeringComplete refusal
Qwen3-8B10.57%0%
Gemma-4-31B5.14%0%

Neither model completely refused to answer any reviewed question. These are aggregate sample statistics, not full-test-set or per-dataset estimates. Together with Appendix A.2, they provide evidence that the trained models retain standard QA capability and do not simply refuse all questions. The external QA evaluation complements, but does not replace, a matched zero-flaw CREDO control.

W2. Why limit the tool rather than evaluate flaw resolution?

The tool primarily lets us observe whether a model proactively attempts cross-verification after noticing credibility concerns, rather than directly accepting the document's answer.

To control the external context, the tool is not connected to the live web. Its first call returns fixed, relevant but inconclusive background material; subsequent calls provide no additional evidence. This excludes variation from search rankings, webpage updates, and incidental access to conclusive answers. All models therefore face the same information conditions.

This is a controlled behavioral experiment, not an evaluation of open-web verification success. It tests whether models seek corroboration and communicate concerns when sufficient corroboration remains unavailable. We also analyze query content rather than treating call frequency as evidence of verification:

ModelSEARCH (%)Question-restating queries (%)IDENTIFY after search (%)
Llama-3.199.999.00.0
Claude Opus 4.858.25.362.4

Question restatement is measured per query; identification after search is conditional on searched instances. These descriptive statistics do not estimate the causal effect of searching. They show why high search frequency alone does not establish credibility-driven cross-verification. Resolving flaws through unrestricted retrieval is a distinct capability not established by this setup.

W3. Why should a model abstain if a flaw does not refute the answer?

We agree that detecting an inconsistency should not automatically require abstention. A flaw does not establish that the target claim is false. A reasonable response can tentatively adopt the document's conclusion, identify the specific concern, express reservations about reliability, and seek independent corroboration. Our goal is not refusal, but answering with uncertainty appropriate to the evidence.

Section 3.2 explicitly separates answer direction from uncertainty: a response that adopts the claim while expressing evidence-related uncertainty counts toward ADOPT but not OVERCLAIM. Thus, qualified adoption can be appropriate. We interpret ADOPT jointly with flaw identification and uncertainty expression, rather than equating lower adoption with better behavior. Identifying specific flaws and communicating reservations, as the reviewer suggests, is precisely the capability we seek to assess.

Anonymous dataset: https://huggingface.co/datasets/anonymous-submission-01/CREDO/tree/main

I thank the authors for their rebuttal. The responses, along with the additional results, do address much of my doubts. I thus raise the score by a point.

Reviewer_ZS7YOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors introduced a term called creadibility-aware evidence reliance, which is something that can answer the question of

Does an LLM tend to be less relying on a claim when other parts of the document have credibility flaws? The authors proposed CREDO benchmark, which contains more than a thousands questions and evaluate many open and propritetary models on it. They found that models can identify a flaw and still adopt a claim.

Critical strengths
  1. Introduced a new research problem/setting and designed benchmark to evaluate models under this new setting.
  2. Large evaluation with consistent results.
Critical weaknesses

There is no control: CREDO does not have a subset that does not contain any flaw. Would be better is we can see the comparison of models' performance on CREDO with flaw and CREDO without flaw.

Questions for authors

What is the closed-book accuracy of each model on the benchmark questions? How often does each model abstain on the seven QA benchmarks?

2 comments

Thank you for your thoughtful review. We address the control setting, closed-book performance, and abstention below.

W1. Evaluation without planted flaws

Existing QA and RAG benchmarks, including NQ, TriviaQA, and HotpotQA, extensively evaluate standard evidence-based answering. CREDO addresses a complementary question: when a document appears to answer the question but contains identifiable credibility concerns, does the model scrutinize it, seek cross-verification, and calibrate its response?

For CREDO's time-sensitive or long-tail questions, establishing both a reliable target answer and the absence of credibility flaws throughout a document requires checking current facts, context, and provenance. This can be difficult when the underlying facts remain uncertain. In contrast, planting a localized, explainable contradiction or error creates a controlled flawed-document instance. This construction asymmetry motivates our focus on flawed conditions.

Our central observation is measured directly in these conditions: many models adopt the document's claim without identifying its flaws or appropriately expressing uncertainty. Appendix A.2 and Figure 7 provide a complementary evaluation on seven external QA benchmarks under the same retrieval setup and Wikipedia index. These assess whether evidence-reflection SFT preserves standard retrieval-based answering, rather than duplicating that task within CREDO. They are complementary evidence, not a matched zero-flaw CREDO control.

Q1. Closed-book accuracy

We understand the concern about possible conflicts between parametric knowledge and the supplied document. Section 2.2 excludes questions judged reliably answerable without retrieval and selects questions according to long-tail-ness, temporal variability, and specificity. This reduces reliance on memorized answers and focuses evaluation on the supplied evidence.

Per-model closed-book accuracy is not reported in the current paper. When a model cannot answer from its parametric knowledge and has no retrieved evidence, the expected behavior is to acknowledge this limitation and say "I don't know." We regard recognizing the limits of parametric knowledge as an important but distinct research topic. CREDO instead studies how models assess and use an answer-bearing document when its credibility is questionable.

We will add this experiment to the appendix by removing the documents and disabling search. We will report abstention and uncertainty-expression rates, together with accuracy on questions whose answers can be independently verified, to better distinguish the contributions of parametric knowledge and retrieved evidence.

Q2. Abstention on the seven QA benchmarks

To complement substring EM, we sampled 100 questions from each of the seven datasets, totaling 700 questions, and manually reviewed the responses from both models. The reviewed checkpoints are Qwen3-8B and Gemma-4-31B after evidence-reflection SFT, not their base models. We distinguished uncertainty expressed while answering from complete refusal to answer.

Model after evidence-reflection SFTUncertainty while answeringComplete refusal
Qwen3-8B10.57%0%
Gemma-4-31B5.14%0%

These are aggregate statistics over the reviewed samples, not full-test-set or per-dataset estimates. Neither model completely refused to answer any reviewed question. Thus, uncertainty expressions should not automatically be counted as abstentions. The substring EM results in Appendix A.2 separately show that standard QA performance is preserved relative to the base checkpoints.

The dataset is available anonymously at: https://huggingface.co/datasets/anonymous-submission-01/CREDO/tree/main

Thank you for your rebuttal and new experiments. I will keep my rating for acceptance.

Comments on the paper

3 comments
Reviewer_LzweOpenReview ↗

Followup Questions and Opinions #1

Thank you for the response. It does not change my assessment: the core question in W1/Q1 has not been answered, and the concrete questions in Q2/Q3 have not been answered either. I set out the reasons below and ask the AC and the other reviewers to note them. I also ask the other reviewers to check whether their own questions were actually answered.

1. W1 / Q1: the two readings are never separated

My Q1 was a factual question: are the flaws located outside the target evidence (A), or in the support chain of the conclusion (B)? This is not a framing the authors may choose freely; it is two incompatible construction descriptions written in the paper itself. For the AC's convenience, I list the relevant statements under each.

Statements belonging to Reading A

  • Abstract, Section 1 (p. 2), Section 2.2 (p. 3), Section 5 (p. 9): documents "retain the target evidence"; the flaws are "outside the evidence relevant to the target claim"; none "directly contradict" that claim.
  • p. 1: the flaws "warrant caution in relying on its target evidence, even if the target claim is true."
  • The example in the authors' follow-up: the error "may not be directly related to the claim that the person is a spy" and only makes the source look unprofessional.

Statements belonging to Reading B

  • A.7.3 synthesis prompt: the document "whose critical support chain contains one subtle gap"; the generator must "choose one conclusion-critical bridge," put "the assigned and only gap" in that bridge, and copy the cue span "from the weak premise before the final claim."
  • A.7.3 review prompt: once the cue-bearing material is removed, the remaining body must not independently entail the conclusion.
  • A.7.3, same paragraph: "The gap may weaken the current claim but must not prove the opposite answer" — this describes a gap weakening the claim's support, not a flaw unrelated to the claim.
  • The paper's examples: in Fig. 9 the inconsistent Slovenian yields and conflicting fulfillment totals sit on the data needed to answer; in Fig. 1 the conflicting trial-operation dates bear directly on the opening date.

This is not a wording slip. The main text says the flaws lie outside the target evidence and do not directly contradict the claim; the appendix's construction and review prompt require the flaw to sit in the conclusion's support chain, and the remaining body to not independently entail the conclusion once the cue is removed. These are two different tasks and cannot both hold. The response mixes sentences from both readings: it uses a Reading-B sentence ("weaken the current claim") to argue a Reading-A position, and does not address the one B sentence that discriminates between them. The mixture allows the response to cite B when pressed on A, and to return to A when pressed on B. If the authors believe the two readings can coexist, they should point to the sentence in the paper that makes them compatible.

In the same response, the authors attribute the disagreement to a reader's "fundamental misunderstanding" while conceding that "the non-refutation requirement alone does not resolve the separate question of where flaws lie relative to the support chain," and promising to revise the construction description. Both cannot be true: if it were only a reader's misunderstanding, the description would not need to change; if it needs to change, the problem is in the paper.

The example in the authors' follow-up comment indicates that they in fact adopt Reading A. If so:

  • Under Reading A, identifying a flaw and still adopting the claim is not a failure. The authors make the same point in their reply to Reviewer LtiJ: qualified adoption is appropriate, "rather than equating lower adoption with better behavior." But the title, the TL;DR, the abstract, and the "85%–99% adopt" / "even when they identify those flaws" framing all treat adoption as the failure, which holds only under Reading B.
  • Reading A therefore requires rewriting the title, abstract, introduction, and main conclusion. That is a redesign, not a revision.
  • If the authors confirm Reading B, the paper's data description conflicts with A.7.3, and the task overlaps with evidence-sufficiency detection, so the novelty claim fails.

The authors state that the problem will be defined in detail in the final paper. A benchmark whose core construct can only be defined in the camera-ready has no evaluable construct in this submission. This is neither something the discussion period can resolve nor something a revision can resolve.

The example also replaces CREDO's factual questions with an unverifiable anonymous allegation. The prior for such an allegation is very low, which makes "do not fully accept" look obvious; it provides no normative basis for ADOPT on CREDO's questions, and does not explain why lower adoption should be the improvement direction.

Reviewer_LzweOpenReview ↗

Followup Questions and Opinions #2

2. Q2 / Q3: the concrete factual questions were not answered

  • Q2's language question is not addressed at all: in what language are the data constructed and the models evaluated? This is not a detail that can wait for release: if the data are Chinese and the evaluation English (or the reverse), the interpretation of all 170,000+ trajectories has to be reconsidered.
  • The conflict between "each document has one primary flaw" plus the MULTIPLE_DEFECTS rejection code and "three flaws per document" is not explained. The response restates Section 2.2 and adds that "Table 4 distinguishes five top-level categories from finer construction labels," but does not say why the review prompt presupposes a single flaw, or why three-flaw documents were not rejected.
  • Q3's mapping of A, U, and F is still only "can map" and "provide information for." That is a promise of a mapping, not a mapping. The authors confirm that human validation covers raw judge fields only, so the problem stands.

On the correspondence between the response and the paper's actual procedure: Q2 and Q3 are short-answer questions — the mapping of A, U, and F is a few formulas, the evaluation language is one sentence, and the flaw count is a number the paper itself states in Section 2.2. The response gives restatements and promises rather than any direct answer.

I do not speculate about how the response was produced. But a response written from the paper's actual procedure should have been able to state these lines in two sentences. They do not appear. There are only two explanations: the response was not written from the actual procedure, or the actual procedure is not yet determined. Either affects soundness.

3. For the AC's and the other reviewers' attention

  1. The core factual question in W1/Q1 was not answered, and the position the response actually adopts (Reading A) is incompatible with the paper's main conclusion. This is not a clarification issue; it is a self-contradiction in the paper's core construct.
  2. Q2/Q3 were not answered. On the extraction rules for A, U, and F — a few lines of formulas and a small mapping table — the response says nothing.
  3. The pattern above leaves me unable to confirm that the actual procedure matches the paper's description; I ask the AC to consider this together with the paper's internal inconsistencies.
  4. The follow-up introduces the example as "a timely example," asks whether a named living individual had an affair with a Chinese female spy, and in the next paragraph treats "this person is a spy" as the claim to be evidenced. I consider this to conflict with the Scope clause of the ICLR Code of Conduct and with the Avoid Harm, Respect Privacy, and Be Fair and Take Action not to Discriminate provisions of the ICLR Code of Ethics; it places an individual unconnected to the paper in a national-security context, and the authors should never have introduced it. This requires separate handling by the AC and the program chairs; the provisions and requests are set out in my separate comment and ethics flag, and I do not expand on them here.
  5. Key questions raised by the other reviewers, in particular the zero-flaw control, were likewise not actually answered in the response. I ask the AC to take this into account, and the other reviewers to re-check their own questions.

On the above, what is required is a redesign, not a revision. My rating remains 1, and I recommend rejection.

AuthorsOpenReview ↗

Public Apology and Retraction of Earlier Follow-up Comment

Dear Reviewer Lzwe, Area Chair, and Readers,

We sincerely apologize for the inappropriate example included in the earlier follow-up comment (note 9rgESimevH). Using a named living individual in an unsubstantiated hypothetical scenario involving sensitive allegations was a serious lapse in judgment and inconsistent with the ICLR Code of Conduct and Code of Ethics.

We have deleted that comment to withdraw the example in full. We do not stand by its content and guarantee that neither this example nor anything similar will appear in any form in the manuscript or revisions.

We thank Reviewer Lzwe for bringing this issue to our attention and to the attention of the chairs. The retracted comment was posted hastily without prior review by the full author team. We take this oversight seriously and have put strict team-wide review safeguards in place for all further communications.

Sincerely,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.