- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper focuses on two approaches to belief maintenance in LLM reasoning: editing and rebuilding. Editing revises existing conclusions and their supporting inferences, whereas rebuilding rederives new conclusions from the initial facts and rules under changed assumptions. To compare these two approaches, the paper conducts experiments within a temporal framework that simultaneously records the ordered process of rule application and the current belief values.
The paper introduces NMR-RETRACTION to evaluate the modifications required and the reasonable retention involved in each maintenance procedure. Formal semantics benchmarks are used to provide symbolic reference answers for belief computation that support target beliefs and their derived conclusions. Existing models do not consistently accept counterfactual assumptions, and their average accuracy in predicting the impact of retraction behavior on related beliefs is low. Further analysis indicates that it is difficult to use retained reasoning history to determine which beliefs should be revised or retained, and that supervised fine-tuning can improve models’ belief maintenance capability.
Critical strengths
The paper introduces a framework for nonmonotonic belief maintenance in LLM reasoning based on support dependencies and ordered rule-application histories, which helps researchers better understand how the LLM retains or changes its beliefs.
Critical weaknesses
Regarding problem modeling, the explanation of the modeling process is not sufficiently clear, making it difficult to fully understand. In terms of experiments, the evaluation of the experimental methods lacks cross-method comparison, and the experimental evaluation process lacks details; for example, no specific evaluation criteria are provided for the counterfactual acceptance rates and independent retraction accuracies of the various models and methods.
Questions for authors
In what aspects do you think your method is superior to other related research methods?
3 comments
Weakness 1. Ambiguity in the research setting
We begin with grounded beliefs and rules that produce an inference chain:
\qquad A \xrightarrow{r_{AB}} B \xrightarrow{r_{BC}} C.$$ Here, $A$ is derived from the grounded beliefs, $B$ from $A$, and $C$ from $B$. A counterfactual changes $B$’s value; we use $B'$ as shorthand for $B$ with its newly assigned value. Under explicitly supplied support cuts and maintenance instructions, we examine three operations: | Operation | Evaluation question | |---|---| | Forward retraction | When $B$ is withdrawn, is $C$ also withdrawn? | | Backward retraction | When $B$ is withdrawn, is its support from $A$ also withdrawn? | | Counterfactual propagation | When $B$ changes to $B'$, is $B'$ correctly propagated? | For this example, the specified cut removes $r_{AB}$, and no alternative support preserves $C$. The two maintenance procedures can be illustrated as follows: - **Rebuild:** Start from $\mathcal G\cup\{B'\}$, use $\mathcal R\setminus\{r_{AB}\}$, and recompute. - **Edit:** - withdraw $C$’s support from $B$ - withdraw $B$’s support from $A$ - replace $B$ with $B'$ - propagate $B'$. - **Theoretically**, we ask which propositions receive the same values under Rebuild and Edit, and which may receive different values. Our analysis characterizes conditions for agreement and explains how differences in retained support histories can lead to different outcomes, including under subsequent retraction. - **Experimentally**, we evaluate whether LLMs correctly predict these proposition values under the specified procedure, using symbolic reference labels. ---- **Weakness 2. Insufficient detail on evaluation procedures and metrics** **1. Independent retraction evaluation** Forward and backward retraction are evaluated independently, with the correct preceding state supplied. We score the queried belief values immediately after the designated operation, before subsequent inference. This isolates retraction accuracy from errors in earlier stages. **2. Counterfactual preservation and propagation** After assigning $B'$, we evaluate whether the model preserves this value through subsequent reasoning and correctly determines other belief values. | Component | Evaluation question | |---|---| | CF preservation | Does $B$ retain its assigned value $B'$ after subsequent reasoning? | | On-CF | Are beliefs reached by propagation from $B'$ correctly updated or preserved? | | Off-CF | Are beliefs outside that propagation path correctly updated or preserved? | The CF target is excluded from the On-CF and Off-CF evaluations and scored separately. Importantly, Off-CF does not mean unaffected: withdrawing support can change beliefs outside the CF propagation path. Conversely, a proposition reached by CF propagation may retain its original value despite a change in support. We therefore distinguish required value changes from justified preservation in both regions. **3. Evaluation metrics** Each queried proposition has a symbolic reference value $y_i\in\{\mathrm T,\mathrm F,\mathrm U\}$ under the specified procedure and evaluation stage. Here, $\mathrm U$ denotes an undetermined value. Let $\hat y_i$ be the model prediction and $b_i$ the baseline value. - **Belief Update (BU):** Does the model correctly change a value that should change? - **Belief Maintenance (BM):** Does the model correctly preserve a value that should remain unchanged? $$\mathcal Q_{\mathrm{BU}}=\{i:y_i\neq b_i\}, \qquad \mathcal Q_{\mathrm{BM}}=\{i:y_i=b_i\}.$$ For either subset, we measure exact-match accuracy: For either subset, we measure exact-match accuracy: $$\operatorname{Acc}(\mathcal Q)=\frac{1}{\operatorname{card}(\mathcal Q)}\sum_{i\in\mathcal Q}\mathbf{1}[\hat y_i=y_i].$$ Here, $\operatorname{card}(\mathcal Q)$ denotes the number of evaluated queries. For final-state evaluation, $b_i$ is the initial belief value. For independent retraction diagnostics, $b_i$ is the value immediately before the evaluated operation. Counterfactual preservation measures whether the final prediction at the CF target matches its assigned value $c_i$: $$\operatorname{CFPres}=\frac{1}{\operatorname{card}(\mathcal Q_{\mathrm{CF}})}\sum_{i\in\mathcal Q_{\mathrm{CF}}}\mathbf{1}[\hat y_{i,\mathrm{final}}=c_i].$$ Cross-procedure comparisons distinguish matched queries with identical reference answers from queries scored against each procedure’s own reference answer.Based on our theoretical framework and experimental results, we make the following empirical observations:
-
Belief updates remain difficult. Models preserve unchanged values more accurately than they perform required retraction and revision, even with explicit instructions.
-
Rebuild and Edit have different strengths. Edit generally performs better on backward retraction and shared-gold Off-CF updates; Rebuild performs better or comparably on On-CF updates and better on Off-CF preservation.
-
SFT improves belief revision. Direct-answer supervision yields substantial gains that transfer to CoT and MultiTurn prompting.
We will make these evaluation stages and metric definitions explicit in the revised manuscript.
Response to Questions for Authors
We thank the reviewer for their careful review and positive assessment, and for highlighting where our problem formulation and experimental evaluation need clearer explanations.
What advantages does your approach offer over related methods?
We thank the reviewer for this question. Related work approaches belief change from different perspectives. LLM knowledge editing examines whether changing a fact (e.g., “France’s capital is Paris” → “France’s capital is Rome”) produces consistent changes in semantically related knowledge. Symbolic belief revision and defeasible reasoning study how beliefs and conclusions change when facts, rules, or their justifications are withdrawn.
Our work focuses on a practical question from the perspective of a user observing an LLM’s reasoning: when a particular inferred belief is withdrawn or counterfactually changed, which parts of the existing reasoning should be revised, and which should remain valid? A user may be interested in a particular conclusion and its consequences without needing to inspect the entire belief set. Our framework supports this selective examination by making the relevant support dependencies and maintenance operations explicit. Its advantage lies in connecting a user’s targeted revision request with the operations required to carry it out and their verifiable semantic consequences.
We formalize Rebuild and Edit to provide procedure-specific reference labels for the queried beliefs. These labels distinguish counterfactual acceptance from correct downstream revision and preservation, while revealing when the two procedures yield different outcomes.
We believe our formulation provides a precisely specified starting point for examining practical belief maintenance and extending evaluation to broader representations and alternative policies for retraction and counterfactual revision. Such extensions are important because our rule-based semantics do not capture every form of belief maintenance in LLMs: beliefs may involve uncertainty, priorities may differ from recency, and some reasoning may not be fully expressible through propositions and rules.