acceptodds
Under review as a conference paper at ICLR 2027

When Code Changes, Answers Don't: Breaking Semantic Inertia in Code Models

13 views

Abstract

Code models increasingly reason about programs that change, yet their predictions often remain anchored to what the program did before an edit. We call this failure semantic inertia. A small edit may change the output for one input while preserving it for another, so reliable reasoning requires a selective update rather than a blanket reaction to modified code. Across three execution-tuned Qwen3 models, 93% of edited-program errors following a correct original prediction repeat the original output. On a separate frozen panel of 15 models and checkpoints, semantic inertia spans model families and remains measurable even in the strongest hosted models. We introduce the Semantic Inertia Benchmark to measure this capability directly. Each group pairs an original and edited program with a Changed input that requires Revision and an Unchanged input that requires Preservation; Joint accuracy requires all four program–input predictions to be correct. We then propose Counterfactual Regrounding, which trains on these execution contrasts, emphasizes edited-program answers, and retains correct original-program judgments. Inference remains a single forward pass on one program and input. On 1,000 held-out APPS-derived groups, Counterfactual Regrounding raises Qwen3-4B's Joint accuracy from 9.5% to 16.0%, lowers inertia from 75.4% to 63.5%, and improves Preservation. Its gains grow with Qwen3 model size, transfer to OpenCoder-1.5B, extend to unseen edit families and open-ended generation, and persist on the separate 2,323-group panel. These results establish selective semantic updating as a distinct capability—and execution contrasts as an effective way to train it.

Primary area:
interpretability and explainable AI
Keywords:
Large Language Models, Code Reasoning, Program Execution, Code Edits, Semantic Inertia, Counterfactual Learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_Z4u9OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies semantic inertia in code language models, where models predict the original program's output despite behavior-changing edits. The authors introduce the Semantic Inertia Benchmark, which evaluates both Revision on inputs whose outputs change and Preservation on inputs whose outputs remain unchanged. To address this issue, they propose Counterfactual Regrounding, which combines execution-contrast training data, weighted supervision on edited programs, and KL-based retention. Experiments across several code models and datasets show improvements in Joint accuracy and reductions in semantic inertia.

Critical strengths
  1. The paper investigates an interesting problem of how code models respond to program modifications.

  2. The joint evaluation of Revision and Preservation provides a useful perspective on model behavior under code edits.

Critical weaknesses
  1. Validity and Necessity of Semantic Inertia: From my understanding, the original and edited programs in this paper are evaluated independently, meaning that the model neither observes the edit nor has any previous prediction to update. Therefore, predicting the original output for an edited program does not necessarily indicate that the model is anchored to a previous semantic. Such errors could simply arise from incorrectly interpreting or executing the modified program; please correct me if I misunderstood anything. Moreover, the proposed Joint accuracy only requires four independent execution predictions to be correct. It remains unclear what additional reasoning capability this benchmark evaluates beyond existing program execution benchmarks.

  2. Limited Conceptual Novelty: The key motivation of leveraging program modifications that preserve execution behavior for some inputs while changing it for others has been extensively studied in compiler testing, particularly through Equivalence Modulo Inputs (EMI) [a] and its subsequent work [b, c]. These studies have already explored input-dependent semantic equivalence and systematically investigated program mutations that preserve execution behavior under selected inputs, including mutations to executed code. More directly, [d] has already applied semantics-preserving program mutations to evaluate LLM code reasoning, demonstrating that models can fail to maintain correct execution predictions even when program semantics remain unchanged. Although this paper combines Changed and Unchanged inputs to evaluate code models, the underlying semantic relationships and testing principles are already well established. The paper needs to clarify what fundamentally new challenge or insight its formulation introduces beyond adapting these existing ideas to program execution prediction and the previously identified sensitivity of code models to program mutations

  3. Unclear Methodological Novelty: The Counterfactual Regrounding seems not explicitly utilize the counterfactual relationship between original and edited programs. Although the training examples are constructed from program pairs, the objective consists of independent cross-entropy losses with increased weights for edited-program examples and KL regularization on original-program predictions. Neither component explicitly models the semantic relationship between the two program versions. As a result, the same objective can be optimized using the individual training examples without retaining their pairing relationships. The Original-only baseline does not address this concern because it also removes edited-program supervision, making it impossible to isolate the contribution of counterfactual relationships from additional execution training. The paper needs to clarify what Counterfactual Regrounding contributes beyond conventional supervised fine-tuning with augmented execution examples.

  4. Alternative Explanation for Semantic Inertia: The benchmark constructs original programs from existing code datasets and generates edited versions through local operator replacements. Since the original programs generally follow familiar algorithmic patterns, the edited versions may exhibit behavior that conflicts with patterns learned during pretraining. For example, if a function named count_positive originally uses x > 0 and the condition is modified to x >= 0, a model may still count only strictly positive values because this behavior aligns with the function's intended purpose and common implementations. Such an "incorrect" prediction is not due to the preservation of old semantics. Similarly, training on mutated programs may reduce these errors simply by exposing the model to implementations that deviate from familiar patterns. Since the experiments do not control for this asymmetry between original and edited programs, they cannot distinguish whether the observed errors arise from the proposed semantic inertia mechanism or learned preferences for familiar program behavior.

References

[a] Compiler Validation via Equivalence Modulo Inputs. In PLDI 2014.

[b] Finding Deep Compiler Bugs via Guided Stochastic Program Mutation. In OOPSLA 2015.

[c] Finding Compiler Bugs via Live Code Mutation. In OOPSLA 2016.

[e] Is Your Benchmark (Still) Useful? Dynamic Benchmarking for Code Language Models. In DL4C @ NeurIPS 2025.

Questions for authors
  1. Given that the original and edited programs are evaluated independently, what distinguishes semantic inertia from ordinary program execution errors, and what additional reasoning capability does the proposed Joint metric capture?

  2. What fundamentally new testing principle or semantic reasoning challenge does the proposed benchmark introduce beyond existing EMI-based compiler testing and mutation-based evaluation of code language models?

  3. Since the training objective consists of independent weighted cross-entropy losses and KL regularization, what does Counterfactual Regrounding learn from the pairing relationships beyond conventional supervised fine-tuning on augmented execution examples?

  4. How do the authors distinguish semantic inertia from models' learned preferences for familiar program behavior, given the asymmetry between original implementations and modified programs?

Reviewer_vKmgOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper defines semantic inertia after code edits and introduces a benchmark with Changed and Unchanged inputs. It proposes Counterfactual Regrounding, which trains on execution contrasts, upweights edited-program answers, and applies KL retention on correct original-program predictions. Experiments cover Qwen3 models, OpenCoder, unseen edit types, open-ended generation, and a frozen model panel.

Critical strengths
  • The Changed/Unchanged formulation is clear and distinguishes selective semantic updating from indiscriminate answer changes.
  • The benchmark labels are execution-derived, the four questions are answered independently, and the method, development, and model-panel tracks are separated.
  • The method remains a single-forward-pass procedure at inference time. The authors also provide several controls, including Appendix F interface checks and a hosted-model protocol.
Critical weaknesses
  • The method's Joint advantage over strong matched baselines is not statistically clear. Counterfactual Regrounding reaches 16.0 Joint accuracy, while Counterfactual SFT reaches 15.1, Revision-only reaches 15.3, and a higher-learning-rate Counterfactual SFT reaches 16.3. Table 11 reports Counterfactual SFT minus the proposed method as −0.8 [−2.3, 0.7], so the proposed method is +0.8 [−0.7, 2.3] relative to SFT; the interval crosses zero. The higher-learning-rate SFT is not Pareto superior, however: its Preservation is 73.7 versus 79.6 and its spurious-switch rate is 10.6 versus 6.0. The evidence supports a selective-update trade-off, but not a clear Joint advantage; a pre-specified combined criterion is needed.
  • Benchmark validity still needs stronger support. The evaluation comes from the public APPS training split. Section A.4 reports a code-shingle overlap check, but there is no model-level pretraining-corpus audit. A post-cutoff or private benchmark, or a stronger contamination analysis, is needed before the benchmark can be treated as contamination-free.
  • The open-ended and interface evidence needs to cover the full capability. Open-ended generation is evaluated only on Changed pairs, so it does not measure Preservation or Joint accuracy. The appendix includes useful candidate-order and label controls, but they should be extended to the primary held-out set. Although the paper reports intervals for the main APPS comparisons, Qwen3 scale comparisons, and selected unseen-edit contrasts, some interface, open-ended, and inertia-by-edit-type slices remain unquantified.
  • Novelty and artifact evidence are limited. Code over Words (Xu et al., 2026) already uses the term “semantic inertia” and trains on counterfactual pairs in a different domain, while CodeFort-style and edit-oriented objectives are closer code-editing baselines. The paper should state what is algorithmically new beyond applying execution contrasts to code edits, and provide a code/data manifest. In addition, inertia is conditioned on each model’s own correct-original cohort; the paper already reports paired stale-error intervals and an aggregate diagnostic inertia interval, but the conditional inertia/stale-error results should be reported by slice with the same paired uncertainty.
Questions for authors
  1. Can the authors run a factorial comparison against matched baselines on a contamination-free held-out set, using a pre-specified combined utility or Pareto criterion?
  2. Can they provide complete open-ended Preservation/Joint results and extend the interface checks to the primary held-out set?
  3. Can they release the benchmark artifact, a code/data manifest, and paired uncertainty for the specific interface, open-ended, and inertia-by-edit-type slices that remain unquantified?
Reviewer_HpNoOpenReview ↗
Rating
2
Confidence
4(+1)
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies semantic inertia, where a code model still returns the original output after an edit changes the program’s correct output. It also proposes a benchmark that pairs Changed inputs with Unchanged inputs. The metric requires all four predictions for the original and edited programs to be correct.

The paper further proposes Counterfactual Regrounding, which trains LoRA adapters on these execution contrasts. It gives edited-program answers twice the weight of original-program answers and uses KL regularization to retain previously correct original-program predictions. The method improves both Revision and Preservation.

Critical strengths
  1. The findings in the paper are somewhat interesting. It studies a particular case of program semantics. It evaluates whether models revise their answers when the output changes and preserve them when it does not. The four-question evaluation provides a more fine-grained assessment of reasoning about code edits than evaluating execution accuracy on individual programs, as in prior work. The paper also tests on models beyond small-scale Qwen models.
  2. The training evaluation is comprehensive and carefully designed.
Critical weaknesses
  1. The benchmark has limited coverage. It focuses on local edits to executable Python programs.
  2. As I understand it, the authors aim to use counterfactual training to improve models’ understanding of program semantics. Although the paper evaluates unseen edit types and open-ended output prediction, it would benefit greatly from evidence that the training generalizes to other semantic reasoning tasks, such as program repair or code transformations. Without such, it is difficult to tell whether the gains reflect a broader improvement in semantic reasoning.
  3. The training experiments are limited to small models. Although the paper evaluates stronger models in benchmark comparison, it does not train these models with the proposed method, which leads to weakness 4.
  4. Several SOTA open-source models already show much higher performance and much lower inertia. Table 21 reports Joint accuracy of 84.0% for Kimi-K3 and 82.0% for DeepSeek-V4.1-Flash, while several models have inertia rates below 3%. Given that, it’s reasonable to assume the even-more-capable SOTA closed-source models would achieve higher.
Questions for authors
  1. Can the authors show whether counterfactual training improves broader semantic reasoning tasks?
  2. Can the authors apply the proposed method to stronger models that already achieve high accuracy and low inertia?

I’m happy to raise my score if my concerns can be addressed.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.