acceptodds
Desk rejected at ICLR 2027

Not Whether, But Which: Localizing Concept-level Knowledge Gaps behind LLM Inconsistency

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Large language models (LLMs) exhibit inconsistency: semantically equivalent inputs elicit contradictory factual outputs, eroding their reliability in downstream applications. Prior work has documented this phenomenon but leaves a more fundamental question unanswered: which knowledge deficiencies produce inconsistency? Knowing that a model is inconsistent is of limited use unless we know which knowledge it lacks. In this paper, we posit that inconsistency stems from identifiable, localizable gaps in concept-level knowledge. To test this hypothesis, we introduce KnowledgeMap, a pipeline that pinpoints such gaps by automatically parsing inputs into lexical concepts, applying calibrated perturbations, and then dynamically profiling concept-level proficiency via consistency measurements. Evaluated across large-scale datasets and diverse LLMs, KnowledgeMap reliably isolates specific knowledge deficiencies and yields actionable guidance for targeted model training.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Inconsistency, knowledge evaluation, large language models.

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_Xb6cOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

KnowledgeMap identifies concept-level knowledge deficiencies behind LLM inconsistency via three components: a Knowledge Extractor (lexical concepts with importance weights), a Knowledge Checker (answer consistency under knowledge-invariant perturbations), and a Knowledge Updater (per-concept proficiency scores). Experiments on ConflictBank and TriviaQA (>200K samples; LLaMA-3.1-8B, Qwen2.5-7B-Instruct, Mistral-7B) show that more popular knowledge (Wikipedia pageviews as proxy) correlates with higher consistency, more concepts per question lowers consistency, and low-proficiency concepts yield lower average consistency. The framework moves beyond measuring inconsistency toward identifying the concepts behind it, offering actionable guidance for targeted knowledge enhancement, though whether proficiency scores capture genuine knowledge deficiencies versus perturbation sensitivity or confounders requires further validation.

Critical strengths
  1. The paper addresses a well-motivated gap by moving beyond measuring LLM inconsistency toward identifying the concepts associated with unreliable outputs.

  2. KnowledgeMap offers a modular, interpretable framework that cleanly separates concept extraction, perturbation-based consistency evaluation, and proficiency updating.

  3. Large-scale experiments (>200K samples, two datasets, three LLMs) plus attention to reproducibility and robustness support the empirical findings.

Critical weaknesses
  1. The central concept-level attribution claim is insufficiently validated, since sample-level consistency scores are assigned to all concepts in a question without distinguishing which concept, interaction, or reasoning error actually caused the failure.

  2. The proficiency score relies on unverified assumptions, notably mapping F-T and F-F to the same value and equating perturbation consistency with knowledge mastery.

  3. The experimental validation is largely indirect and correlational, resting on Wikipedia pageviews as a proxy and on associations that may be confounded by question difficulty, entity type, and answer ambiguity.

  4. The contribution over existing perturbation-based evaluation is not isolated, as no systematic baselines or ablations show that the Knowledge Updater and its extra parameters are necessary.

  5. The perturbation design may introduce confounds such as format sensitivity, instruction following, and contextual interference, so it is unclear that perturbations isolate knowledge-related inconsistency.

  6. Evidence for practical utility is incomplete, since downstream benefits such as targeted fine-tuning are proposed rather than demonstrated against random or popularity-based selection.

  7. Generalizability is limited to three small open-source models and two Wikipedia-linked factual QA datasets, leaving larger models, non-Wikipedia knowledge, conversation, and multi-step reasoning unaddressed.

Questions for authors

N/A

Reviewer_xZQSOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper argues that LLM inconsistency on semantically equivalent inputs stems from identifiable gaps in concept-level knowledge, and introduces KnowledgeMap to locate these gaps. KnowledgeMap uses an LLM to parse each question into weighted Wikipedia-style concepts, scores each sample by the strongest perturbation under which the model remains correct, and aggregates these scores into a per-concept proficiency estimate via an adaptive online update. The authors evaluate three 7B-8B open models on ConflictBank (MCQ, format perturbations) and TriviaQA (SAQ, rewrite plus irrelevant context), using Wikipedia pageviews as a proxy for training exposure. They report that consistent samples involve more popular entities, that consistency declines as questions involve more concepts, and that samples containing low-proficiency concepts are 22.77% less consistent.

Critical strengths

S1. Well-motivated question. Moving from whether LLMs are inconsistent to which knowledge makes them so is a useful reframing; a concept-level diagnostic would directly inform targeted data collection and fine-tuning.

S2. Clear, modular pipeline. The Extractor, Checker, and Updater are separable and specified in enough detail to reimplement, with code and all prompts released (Figs. 8-11).

S3. Scale and proxy checks. The study spans over 200k samples, two answer formats, and three models. The pageview proxy is tested for aggregation method (Fig. 5) and for rank stability across five years (Spearman ρ > 0.8, Tab. 6).

S4. The decisive comparison is present. Among originally correct samples, T-T samples involve significantly more popular entities than T-F samples in 5 of 6 model-dataset pairs (Tab. 3). This is the cleanest evidence in the paper that robustness, and not only accuracy, tracks exposure, and it deserves more prominence.

Critical weaknesses

W1. The consistency score is largely an accuracy score. Because F-T and F-F both score 0 (Sec. 4.4), s = 0 for every sample answered incorrectly on the original input, regardless of its behavior under perturbation. For LLaMA on TriviaQA (62.27% accuracy, Tab. 9), ~38% of scores hence reduce to accuracy. The method also records correctness rather than answer agreement, so a model that repeats the same wrong answer is indistinguishable from one that changes it. The headline popularity result therefore largely restates the known accuracy-popularity relationship (Mallen et al., 2023; Kandpal et al., 2023, not cited). Scoring F-T as 0 also contradicts Secs. 3.2 and 5.1, which describe F-T as partial knowledge.

W2. The validation of concept proficiency is circular. Tab. 4 selects low-proficiency concepts by their low consistency and then shows that their samples are less consistent, so the 22.77% gap follows largely by construction. The Limitations section presents r = 0.58 (Fig. 3, right) as convergent validity against popularity. However, Fig. 3 and App. C.3 describe it as a correlation with average consistency, i.e., the quantity proficiency is computed from. Neither direct test is reported: proficiency against each concept's own pageviews, or proficiency on held-out samples. Tab. 4 also uses the low-proficiency group as the denominator; relative to the "Other" baseline, the changes are -31.95%, -18.56%, and -9.21%.

W3. Perturbation intensity depends on input length. Intensity is 1 minus Sentence-BERT cosine similarity (Sec. 3.2). The same prefix therefore yields lower intensity, and hence lower s_max, for longer questions, independent of the model. Questions with more concepts are longer, so this effect alone could produce part of the decline in Fig. 4. Tab. 9 further shows accuracy dropping 6.8 points from original -> L1 but only 2.1 points from L1 -> L5. This suggests that the LLM rewrite, rather than the added context, drives most of the effect.

W4. The popularity proxy is weaker than presented. The collection window (March 2024 to March 2025) postdates the training data of LLaMA-3.1 and Mistral-7B. Tab. 7 is dominated by 2024 news events (e.g., Kamala Harris, Katie Ledecky, Kris Kristofferson), which undercuts the stability argument in App. B.1.2. The file names in App. B.1.2 follow Wikimedia's hourly naming convention; if the main signal also uses one hour per month, long-tail estimates rest on single-digit counts (cf. Tab. 5). Tab. 3 applies t-tests to raw, heavy-tailed pageviews (Fig. 7) without effect sizes or multiple-comparison correction.

W5. The Updater is unvalidated and inconsistent with its description.

  • With the logistic sigmoid in Alg. 1, v' ∈ [0.5, 1] and r' ∈ [0, 0.5], so η cannot exceed 0.55. Fig. 6 shows values of 0.03-0.24, below η_min = 0.1.
  • θ = 0.31 is calibrated on LLaMA/TriviaQA, whereas ConflictBank intensities implied by Tab. 12 rarely exceed 0.26, so every update there would be negative.
  • Concepts receive 3.99 updates on average (App. C.3) against a window of k = 20, so the adaptive learning rate likely has little effect. No ablation against a simple per-concept mean is reported.

W6. Several reported statistics are internally inconsistent. In 39 of 40 cells of Tab. 13, the mean, standard deviation, Q1, and Q3 equal exactly the values produced by three numbers {min, median, max}. For example, L1 PSim+BERT {0.6938, 0.8402, 0.8885} yields 0.8075 / 0.1014 / 0.7670 / 0.8643, as reported. The remaining cell reports a mean of 0.0000 with a maximum of 0.1051. Fig. 4 also extends to 18 concepts per sample, although CONCEPT_LIMIT = 10 (App. B.2).

W7. The scope is narrower than the claims. Secs. 5.2-5.3 use a single model and dataset (LLaMA, TriviaQA), selected for performance rather than representativeness. Concepts are entities only, so the method identifies which entities co-occur with failures rather than which facts are missing. The abstract and introduction claim benefits for targeted fine-tuning and performance prediction, yet neither is tested.

Questions for authors

Q1. Does the popularity result hold when consistency is defined as answer agreement across perturbations, independent of correctness? How does proficiency compare to a per-concept accuracy baseline? (W1)

Q2. What is the correlation between each concept's proficiency and its own pageviews? Does proficiency estimated on one data split predict consistency on a held-out split? (W2)

Q3. What decoding settings are used, and what is the T-F rate when the identical question is re-asked without perturbation? (W1, W3)

Q4. Which model extracts concepts for each tested model? If each model extracts its own, how are cross-model comparisons kept on a common concept set?

Q5. Were hourly or daily pageview files used in the main analysis? (W4)

Q6. What sample size underlies Tab. 13, and how does Fig. 4 reach 18 concepts under CONCEPT_LIMIT = 10? (W6)

Q7. How is SAQ correctness determined? How is a sample with five perturbations assigned to a single group in Tabs. 2-3? What fraction of responses was excluded for format violations at each level?

Q8. Does fine-tuning on the identified low-proficiency concepts improve held-out consistency relative to a random-concept control? (W7)

Reviewer_7fxYOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper asks which knowledge gaps make llms give inconsistent factual answers. The proposed pipeline, KnowledgeMap, first uses an LLM to split each question into at most 10 concepts with importance weights. It then perturbs each question at five levels, compares the correctness before and after each perturbation, and turns these outcomes into a consistency score. Finally, an update rule raises or lowers a proficiency value for each concept.

Critical strengths

The goal is useful. If we know which concepts a model is weak on, this could help to select training data. The method is simple, and the prompts and parameters are given.

The study is large: two datasets with different formats, three models and five perturbation levels.

Critical weaknesses

The proficiency values don’t seem have an independent test. Proficiency is computed from the consistency score(Algorithm1, eq.2). Figure 3(right) then compares proficiency with the average consistency of the related samples(Appendix C.3), so I think this correlation comes partly from the method itself. Table4 seems to have a similar issue, the low-proficiency concepts are selected by p < 0.4, which comes from consistency of their samples, and then their samples show lower consistency. Only outside signal there is popularity, and its gap is smaller with no significance test, as far as I can see. There is also no simple baseline, for example per-concept accuracy, and no training experiment, although the abstract says that the method gives "actionable guidance for targeted model training". So, the main "which knowledge" claim does not yet seem to have independent support.

Scores mostly measure accuracy. Both F-T and F-F get a case score of 0(Section4.4), so every question that the model first answers wrongly gets a score of 0, whatever happens after the perturbation(eq.1). This also does not fit what stated, which says that F-T may reflect knowledge that exists but is fragile(L205-209). The paper itself describes F-T in two different ways: "no knowledge" in Section4.4, but "ambiguity or partial knowledge" in Section 5.1. Because the paper assumes that popular entities are known better(L302-304), the popularity results may partly restate this assumption through accuracy, The claim that T-T is significantly more popular across all datasets and models (p < 0.001) conflicts with Table 3, which reports non-significant results for Mistral in several comparisons.

Questions for authors

can you show the link between concept proficiency and a value that does not come from the consistency score, for example the pageviews of each concept? Why do F-T and F-F get the same case score of 0, when the text says that F-T may mean fragile but existing knowledge? How do the results change with a non-zero F-T score? Which model extracts the concepts in the main experiments, and with which decoding settings?

Reviewer_EcH8OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes KnowledgeMap, a framework for understanding LLM inconsistency by identifying knowledge gaps at the concept level. It uses an LLM to extract concepts and their importance weights from input questions, applies different levels of knowledge-preserving perturbations to measure response consistency, and updates a proficiency score for each concept. The authors evaluate the framework on ConflictBank and TriviaQA using three LLMs. The results show that lower concept proficiency is associated with lower response consistency, and questions involving more concepts tend to be less consistent. The goal is to identify specific knowledge weaknesses that could guide future model improvements.

Critical strengths

The paper addresses an important problem in LLM reliability. Instead of only measuring whether a model is inconsistent, it tries to identify which knowledge concepts are associated with these failures. This could be useful for understanding model weaknesses and guiding targeted improvements. The evaluation covers two datasets with over 200,000 original samples and three different language models. The paper also examines how knowledge popularity, concept complexity, and response consistency relate to each other, providing a useful set of empirical observations.

Critical weaknesses
  1. A major concern is that concept proficiency is directly updated using consistency scores, but the paper then uses the correlation between proficiency and consistency as evidence that the framework works. Since these two quantities are already linked by design, this result is somewhat expected. The Wikipedia popularity analysis provides additional evidence, but it does not directly validate whether the estimated scores reflect the model's actual knowledge.
  2. The paper makes fairly strong claims that specific knowledge gaps cause inconsistency, but the evidence is mainly correlational. For example, Table 4 shows lower consistency for samples containing low-proficiency concepts, but this does not prove that those concepts caused the failures. Question difficulty, reasoning complexity, and interactions between concepts could also explain the results. A more direct intervention would be needed to support the causal claims.
Questions for authors
  1. Concept proficiency is updated using consistency scores, but Figure 3 uses the correlation between proficiency and consistency to validate the method. Could you evaluate the proficiency scores against an independent measure of concept knowledge?
  2. Table 4 shows a correlation, but other factors such as question difficulty or reasoning complexity may also explain the results. Have you tried controlling for these factors or testing the effect of fixing a specific knowledge gap?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.