- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper studies semantic inertia in code language models, where models predict the original program's output despite behavior-changing edits. The authors introduce the Semantic Inertia Benchmark, which evaluates both Revision on inputs whose outputs change and Preservation on inputs whose outputs remain unchanged. To address this issue, they propose Counterfactual Regrounding, which combines execution-contrast training data, weighted supervision on edited programs, and KL-based retention. Experiments across several code models and datasets show improvements in Joint accuracy and reductions in semantic inertia.
Critical strengths
-
The paper investigates an interesting problem of how code models respond to program modifications.
-
The joint evaluation of Revision and Preservation provides a useful perspective on model behavior under code edits.
Critical weaknesses
-
Validity and Necessity of Semantic Inertia: From my understanding, the original and edited programs in this paper are evaluated independently, meaning that the model neither observes the edit nor has any previous prediction to update. Therefore, predicting the original output for an edited program does not necessarily indicate that the model is anchored to a previous semantic. Such errors could simply arise from incorrectly interpreting or executing the modified program; please correct me if I misunderstood anything. Moreover, the proposed Joint accuracy only requires four independent execution predictions to be correct. It remains unclear what additional reasoning capability this benchmark evaluates beyond existing program execution benchmarks.
-
Limited Conceptual Novelty: The key motivation of leveraging program modifications that preserve execution behavior for some inputs while changing it for others has been extensively studied in compiler testing, particularly through Equivalence Modulo Inputs (EMI) [a] and its subsequent work [b, c]. These studies have already explored input-dependent semantic equivalence and systematically investigated program mutations that preserve execution behavior under selected inputs, including mutations to executed code. More directly, [d] has already applied semantics-preserving program mutations to evaluate LLM code reasoning, demonstrating that models can fail to maintain correct execution predictions even when program semantics remain unchanged. Although this paper combines Changed and Unchanged inputs to evaluate code models, the underlying semantic relationships and testing principles are already well established. The paper needs to clarify what fundamentally new challenge or insight its formulation introduces beyond adapting these existing ideas to program execution prediction and the previously identified sensitivity of code models to program mutations
-
Unclear Methodological Novelty: The Counterfactual Regrounding seems not explicitly utilize the counterfactual relationship between original and edited programs. Although the training examples are constructed from program pairs, the objective consists of independent cross-entropy losses with increased weights for edited-program examples and KL regularization on original-program predictions. Neither component explicitly models the semantic relationship between the two program versions. As a result, the same objective can be optimized using the individual training examples without retaining their pairing relationships. The Original-only baseline does not address this concern because it also removes edited-program supervision, making it impossible to isolate the contribution of counterfactual relationships from additional execution training. The paper needs to clarify what Counterfactual Regrounding contributes beyond conventional supervised fine-tuning with augmented execution examples.
-
Alternative Explanation for Semantic Inertia: The benchmark constructs original programs from existing code datasets and generates edited versions through local operator replacements. Since the original programs generally follow familiar algorithmic patterns, the edited versions may exhibit behavior that conflicts with patterns learned during pretraining. For example, if a function named
count_positiveoriginally usesx > 0and the condition is modified tox >= 0, a model may still count only strictly positive values because this behavior aligns with the function's intended purpose and common implementations. Such an "incorrect" prediction is not due to the preservation of old semantics. Similarly, training on mutated programs may reduce these errors simply by exposing the model to implementations that deviate from familiar patterns. Since the experiments do not control for this asymmetry between original and edited programs, they cannot distinguish whether the observed errors arise from the proposed semantic inertia mechanism or learned preferences for familiar program behavior.
References
[a] Compiler Validation via Equivalence Modulo Inputs. In PLDI 2014.
[b] Finding Deep Compiler Bugs via Guided Stochastic Program Mutation. In OOPSLA 2015.
[c] Finding Compiler Bugs via Live Code Mutation. In OOPSLA 2016.
[e] Is Your Benchmark (Still) Useful? Dynamic Benchmarking for Code Language Models. In DL4C @ NeurIPS 2025.
Questions for authors
-
Given that the original and edited programs are evaluated independently, what distinguishes semantic inertia from ordinary program execution errors, and what additional reasoning capability does the proposed Joint metric capture?
-
What fundamentally new testing principle or semantic reasoning challenge does the proposed benchmark introduce beyond existing EMI-based compiler testing and mutation-based evaluation of code language models?
-
Since the training objective consists of independent weighted cross-entropy losses and KL regularization, what does Counterfactual Regrounding learn from the pairing relationships beyond conventional supervised fine-tuning on augmented execution examples?
-
How do the authors distinguish semantic inertia from models' learned preferences for familiar program behavior, given the asymmetry between original implementations and modified programs?