acceptodds
Under review as a conference paper at ICLR 2027

Beyond Tool–Raw Comparisons: Matched-Control Attribution of Semantic Assistance for Code Reasoning

Abstract

Tool-versus-Raw evaluations show whether semantic assistance helps, but they mix useful content with prompt surface and computation. We develop source-verified matched-control attribution and apply it to 398 Python and Bash families in 6,608 calls. Source-only SemCheck and TraceCheck improve exact execution prediction by 12.2 and 15.9 percentage points over truthful but irrelevant notes. The gains persist under exact provider-token matching and when both arms use identical neutral labels (+10.6 points in Python and +12.0 in Bash). On a separate 44-stage resolved-state stress test for Claude Opus 4.8, GPT-5.6 Sol, and Gemini 3.8 Flash, an exact late-stage snapshot raises accuracy from 38.9% to 74.4% (+35.6 points). Across matched tool-reliance analyses, Claude Opus 4.8 and GPT-5.6 Sol solve 92.5% of the programs unaided, yet a sparse causal error reduces accuracy to 0.0% and all 40 responses reproduce the tool-induced wrong answer; a verification instruction restores 97.5% accuracy without adding input tokens. Same-bank controls detect no average advantage from source localization, rule filtering, or a full trace over sufficient plain state: the full-trace interval bounds any positive gain to +1.0 point, while localization remains less precisely resolved at up to +7.2 points. The central result is actionable: applicable rules and resolved state drive improvement, simpler delivery retains the gains, and models that can solve a program unaided may still propagate an erroneous tool state exactly.

Primary area:
datasets and benchmarks
Keywords:
code reasoning, program semantics, semantic assistance, large language model evaluation, matched-control evaluation, tool-augmented language models, tool reliability

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_pzYaOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates whether improvements from semantic assistance for code execution prediction arise from useful information or from changes in prompt presentation. It introduces source-verified matched controls and evaluates two deterministic analyzers, SemCheck for Python and TraceCheck for Bash. Across 398 program families and 6,608 model responses, the tools improve exact output prediction over matched controls by 12.2 and 15.9 percentage points, respectively. Additional experiments examine resolved intermediate states, annotation placement, rule selection, trace presentation, and the propagation of incorrect tool states.

Critical strengths

S1. The matched-control design carefully separates relevant information from prompt presentation.

The paper compares useful annotations with truthful but irrelevant information while controlling annotation position, format, and length. Replications additionally match provider-native token counts and use identical neutral labels. These controls strengthen the evidence that task-relevant information contributes to the observed improvements.

S2. The evaluation uses executable ground truth and systematic statistical analysis.

The authors validate outputs through repeated execution and independent state-transition simulators, avoiding reliance on LLM judges. The evaluation also includes family-level statistical inference, multiple-testing correction, and unsuccessful responses in the reported accuracy.

S3. The error-injection experiments provide clear evidence of reliance on supplied states.

The distinction between errors on and outside the output dependency path is useful. Checking whether responses match the exact counterfactual wrong output provides stronger evidence of error propagation than reporting an accuracy decrease alone.

Critical weaknesses

W1. The main findings provide limited new insight beyond straightforward expectations.

Most central findings are readily anticipated. Relevant semantic rules should be more useful than irrelevant facts; supplying a correctly computed intermediate state reduces the remaining reasoning burden; an incorrect state can lead to incorrect downstream outputs; and explicitly requesting verification can mitigate this reliance. The experiments quantify these effects, but the paper does not sufficiently establish what substantially new understanding follows from them.

The matched controls improve attribution rigor. However, on the core banks, Raw and Sham differ by less than three percentage points, and the conventional Tool–Raw comparison leads to a similar qualitative conclusion. The paper therefore does not demonstrate an important case in which the proposed attribution changes the conclusion drawn from conventional evaluation. More generally, the input-output analysis establishes intervention effects but provides limited explanation of why the behaviors occur or when the expected pattern breaks down.

W2. The evaluation does not establish relevance to complete coding agents.

The experiments evaluate models receiving code and precomputed annotations under a task contract that excludes execution. They do not evaluate an interactive agent harness in which the model can select tools, execute code, inspect feedback, and revise its answer. SemCheck and TraceCheck outputs are supplied by the experiment rather than requested and checked by the model during an interactive workflow.

This distinction matters for practical relevance. For deterministic programs with available runtimes, an agent may obtain outputs or intermediate states through execution and instrumentation. The reported gains may therefore compensate for capabilities excluded by the evaluation setup. Although such restrictions can support controlled model analysis, the paper needs to establish why they represent an important practical setting and whether the findings remain useful when execution and feedback are available.

W3. Program size and complexity are insufficiently characterized.

The paper reports combinations of 3–5 semantic mechanisms, programs with 11–13 stages, Bash programs with 4–20 stages, and a 44-stage stress test. However, it does not provide per-program or aggregate statistics for lines of code, code token counts, function counts, or structural complexity. A semantic stage is not a standard measure of program size and may contain multiple definitions and statements.

Consequently, it is difficult to assess the actual scale of the evaluated programs or whether the benchmark captures challenges representative of practical code understanding. The authors should report size distributions and representative complete examples for each bank, distinguishing executable code from annotations and token-matching padding.

W4. External validity remains weak because the main evidence comes from self-constructed synthetic programs.

The authors reasonably explain that synthetic programs enable precise counterfactuals and strict matching. However, they do not sufficiently explain why existing code reasoning benchmarks cannot be adapted to support similar controls, or validate the findings on such benchmarks.

The 80 source-anchored replay families provide only limited additional evidence. They replace project dependencies with deterministic stand-ins and thus do not preserve the full demands of understanding code within its original project. Furthermore, Raw, Sham, and Tool all achieve approximately 97.5% pooled accuracy, yielding no measurable average benefit from assistance. The ceiling effect explains this result but leaves the transfer of the central findings to realistic tasks unresolved.

W5. The findings do not yet provide validated guidance for coding agent design.

The recommendation to prioritize correct semantic content over elaborate presentation is reasonable but does not resolve concrete design choices. The paper does not establish how an agent should identify sufficient state, select information to retain, decide which tool results require verification, or balance verification costs against reliability.

In particular, sufficient boundary state for a fixed output prediction may be insufficient for debugging or repair, where the history explaining how an incorrect state arose can be essential. Similarly, the absence of a significant average benefit from rule localization or selection does not establish that these features are generally unnecessary. The rule panel contains only nine rules, and the intervals still allow positive effects of up to 7.2 and 5.6 percentage points for localization and selection. A demonstrated improvement to an actual agent design would make the practical implications substantially more convincing.

Questions for authors

Q1. What practical setting motivates predicting executable program outputs without execution?

Why should this restricted setting determine the design of semantic assistance for coding agents? Do the reported benefits remain when the same model operates in a harness with code execution, feedback, and verification?

Q2. What are the actual sizes and complexity distributions of the programs?

Please report the minimum, median, mean, and maximum LOC and code token counts for each bank, excluding annotations and audit padding. How do semantic stage counts relate to these measurements?

Q3. Why were existing code reasoning benchmarks not used or adapted?

What specific properties prevent constructing matched controls on existing benchmarks? Can the central effects be reproduced on more challenging source-derived programs that preserve relevant project dependencies?

Reviewer_yzzPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper proposes matched control attribution for evaluating “semantic assistance” on short executable programs. The standard tool-versus-raw comparisons show whether adding semantic help assists models in predicting execution states better. The authors propose sham controls, annotations that are true for the code but irrelevant to the answer. They developed two deterministic source code tools, SemCheck and TraceCheck (respectively for Python and Bash), inserting semantic content into the frozen executable, and ran the tool arm against (i) the True but irrelevant “Sham” arm, (ii) the Raw “No-Annotation” arm. SemCheck improves the exact execution prediction by 12.2%, and TraceCheck improve the performance by 15.9%. Additionally, on a 44-stage stress test, a snapshot raises the accuracy from 38.9% to 74.4%.

Critical strengths
  • The matched control attribution design and the evaluation protocol are rigorous.

  • Relevant semantic content archives significant performance gain

  • The evaluated model list is comprehensive and spans multiple frontier model families.

  • The artifact is released to support reproducibility.

Critical weaknesses
  • The paper is difficult to follow, and the presentation is unclear

  • The authors claim in Section 4.2 that semcheck had a positive effect on all three models. However, Table 6 shows that Gemini 3.5 Flash and GPT 5.6 Terra performed exceptionally well, while Claude Haiku 4.5 performed at the floor. The authors didn’t provide insights into why the gap between Haiku vs Gemini and GPT is so substantial.

  • The current evaluation is almost entirely on synthetic data rather than real-world repositories. 

  • The nine constructs per language are declared; it is not stated how their coverage was measured.

Questions for authors
  • Could the authors provide insights into why Claude Haiku 4.5’s performance is near the floor when the other two models achieve more than 80% performance?

  • Could the authors clarify (i) whether a non-ceiling real-repo exists in the evaluation and (ii) what a non-ceiling design looks like? And can the current tools be extended to real-world repos?

  • Could the authors provide details on the benchmark construction, especially how the nine constructs were selected and the rationale behind the selection?

  • Could the authors also provide the cost of benchmark construction and the evaluation process?

Reviewer_9j3GOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper compares the effect of different hints in prompts for improving code reasoning. It creates synthetic benchmarks and evaluates LLMs in a controlled setting. It disentangles factors such as prompt length, relevant tool semantic information, and formatting. The authors find that relevant semantic information improves accuracy, while the specific places to put such information don’t seem to matter. They also show that LLMs can be misled by incorrect partial states but can also do well if given a correct partial state.

Critical strengths

The experiments are very comprehensive. It highlights LLMs are not robust if we perturb the states in coding tasks.

Critical weaknesses
  • I have a hard time following what the paper is arguing. It lacks flow and motivation. For example, in the abstract and introduction, the paper uses a lot of jargon and digs into implementation details without giving any exposition.

  • In the presentation of the experiments, the authors again list a lot of technical details without stating any motivation. In addition, the results are mostly presented in prose with numbers and confidence intervals, which is very difficult to follow.

  • The experiments, although carried out with a lot of scaffolding and statistical testing, seem to be mostly on synthetic settings. It would be interesting to see if the trends hold on realistic coding benchmarks.

  • The results seem pretty unsurprising. Correct partial states / relevant context information could definitely guide the model for better performance. I find it hard to justify “inserting incorrect partial state” for realistic coding scenarios.

Questions for authors
  • Could the authors demonstrate their findings and insights in realistic coding benchmarks?

  • In the incorrect intermediate-state experiment, the models already have 92.5% base accuracy. Would the verification instruction still help on harder tasks where models have poor base performance and genuinely need tool assistance?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.