- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies selective belief revision: updating judgments affected by evidence changes while preserving unaffected ones. It introduces the Evidence–Intervention–Claim (EIC) framework and a controlled hypergraph architecture to examine evidence propagation and autoregressive state readout. The experiments show that propagation benefits can coexist with unstable judgments: changing a preceding state token can substantially alter a subsequent prediction despite unchanged target evidence and graph contributions. The paper also explores training augmentation and endpoint-factorized generation to mitigate this failure.
Critical strengths
- The sensitivity of evidence-based judgments to previously generated state tokens highlights a failure mode that aggregate task accuracy can obscure.
- Matched source-update controls and interventions on preceding tokens, companion evidence, and earlier generated history provide useful evidence about the computational pathways influencing predictions.
Critical weaknesses
- The findings are demonstrated in two model families, both approximately 8B and trained under the proposed protocol. How robust are these findings across other model scales and backbones?
- The introduction moves too quickly to quantitative results before clearly defining the task, establishing its motivation, and explaining the experimental setting. In particular, since endpoint-factorized generation (§6.7) allows the two endpoint states to be assessed separately, what motivates the joint autoregressive formulation? In which practical settings is joint generation preferable, and to what extent does the observed failure generalize beyond this formulation?
Questions for authors
Could the authors apply the single-token intervention from §6.5 to the augmented models in §6.6?
2 comments
Response to Reviewer DuCF (2/2)
W1 — Additional model scales and backbones. On the primary diagnostic (133 claims/97 sources), reference checkpoints give:
| Model | Original partial | Diagonal first / second | Diagonal Joint | Token effect |
|---|---|---|---|---|
| Qwen3-8B | 94.33 | 82.47 / 14.95 | 11.34 | +67.01 pp |
| Llama-3.1-8B | 91.75 | 85.57 / 13.40 | 8.25 | +63.40 pp |
| Qwen3-14B-Base | 91.24 | 84.54 / 6.70 | 3.61 | +69.59 pp |
| Qwen3-30B-A3B-Base | 91.24 | 82.99 / 18.56 | 13.40 | +59.79 pp |
Accuracy columns are percentages. Token effects are forced-full minus forced-partial second-state accuracy at a fixed remaining prefix. Diagonal scores are free-running. The 30B configuration is an NF4 MoE with about 3B active parameters. Its FLD M-Joint is 90.82% versus 77.04% for matched StaticSource.
Across three seeds each for Qwen-8B, Llama-8B and Qwen-14B, plus this MoE checkpoint, all ten exhibit partial-preservation failure and positive token effects. Separately trained NoGraph models reproduce the failure (0% partial-diagonal Joint in both 8B families). The pattern extends across these tested scales and persists without graph-head injection.
W2 — Why evaluate joint generation? When evidence changes, some claims gain or lose justification while others should retain their warranted state. EIC evaluates these two requirements together: both before/after labels must be correct, so agreement between wrong labels is not successful preservation.
A literature assistant could report claim states before and after a supporting study is corrected. A technical-document assistant could summarize which requirements remain satisfied after a test record changes. Both motivate several evidence-grounded judgments in one structured report. They are illustrative applications, not deployment evaluations. We have introduced the task and motivation before the quantitative example in Section 1.
Joint generation produces this report in one decoding call. Separately generated judgments can be assembled into the same report, so the choice is an interface trade-off rather than a task requirement. Across three Qwen3-8B seeds:
| Interface | N1 Joint (%) | Original A / P / M (%) | Pair median latency (s) |
|---|---|---|---|
| Cached Joint | 15.09 | 95.74 / 98.81 / 92.18 | 2.491 |
| Factorized | 59.79 | 93.82 / 98.13 / 89.46 | 4.132 |
| Shared, recomputation | 8.37 | 95.76 / 98.46 / 92.35 | 3.994 |
| Masked, same implementation | 59.67 | 93.73 / 98.36 / 88.44 | 3.985 |
Accuracy entries are three-seed source-macro means. Latency averages per-checkpoint medians for complete endpoint pairs, including both calls for Factorized. N1 changes which necessary premises are missing while both targets remain partial. These development tests edit existing training-held-out FLD evaluation sources. A/P/M retain the original task.
Cached Joint is fastest here. Factorized and Masked increase preservation with measured original-task costs.
Crucially, Shared/Masked retain training coverage, paired grammar, objective and main-decoder forward-token budget. Replacing only other-endpoint state codes gives +51.30 pp N1, at A/P/M costs of 2.03/0.10/3.91 pp. Preservation can therefore improve substantially within the joint format: the experiments identify generated-label visibility as a controllable source of error.
Code permutation, reversed output order and NoGraph controls extend the diagnosis beyond one state encoding, temporal order or graph branch. The evidence concerns controlled, supplied claims and views. The application examples motivate this evaluation rather than establish prevalence in unrestricted evidence revision.
Response to Reviewer DuCF (1/2)
Thank you for the suggestion. Applying the Section 6.5 intervention to all four Section 6.6 augmentation models reveals that augmentation can change the direction and context of label dependence even as preservation improves. Target evidence, graph residual and the remaining prefix are fixed. On the primary diagnostic (133 claims/97 sources):
Joint scores both freely generated states. Forced-prefix columns score the second state conditionally. Effects use unrounded scores.
The original token effect is +67.01 pp. Both Full controls retain large positive effects. Partial10 reverses the effect: a partial prefix now favors partial. Partial1's diagonal effect is not distinguishable from zero, while the same checkpoint retains +53.61 pp with full companion/history. Its improved diagonal robustness is therefore context-dependent.
The original-task comparison shows the corresponding trade-offs (%). A/P/M denote affected, preserved and multihop-affected Joint:
All-partial/code-0 evaluations contain 276 endpoint observations/174 sources and 410/257, distinct from the 133-claim diagnostic. Ten-percent and one-percent controls replace the same 321 and 32 rows, respectively, out of 3,209 training pairs.
Partial10 emits partial/partial for 389/410 no-coverage endpoints. Its M loss is not specific to the reversed token effect, since Full10 also lowers M. Partial1 preserves far more code-0 and affected-claim accuracy. The intervention distinguishes improved preservation from elimination of label dependence. The dose comparison identifies a less disruptive augmentation regime. We have incorporated these results in Section 6.6 and Appendix F (Tables 24 and 34).