- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies whether LLM execution of a fully specified symbolic procedure stays the same when the representation changes but the semantics do not. The authors build EXACTBENCH-RENDER, and a deterministic oracle and a strict parser grade every answer. The primary experiment compares overlapping row/column role labels (A/B) with disjoint labels (P/Q), while keeping the underlying program semantics, lookup tables, initial states, and decoding parameters unchanged. The paper concludes that symbolic-execution reliability depends not only on the underlying computation but also on its textual representation, motivating representation-robustness evaluation alongside conventional accuracy measurements.
Critical strengths
- Good writing and logic chain, figures show the clear results.
- The paper's main strength is its carefully controlled paired experimental design. By changing only the row/column role labels while preserving program semantics, table contents, initial states, and gold outputs.
- It moves beyond demonstrating that one particular label pair performs better than another and reveals a replicated interaction between role labels and value symbols.
Critical weaknesses
- In the crossed design, a pure overlap account predicts that A/B labels improve when the values move to W–Z. They do not: A/B on A–D (overlap) gives 12.5%/14.1%, while A/B on W–Z (disjoint) gives 13.3%/13.3%.
- In a 2×2 design, the “overlap” contrast is aliased with the label×alphabet interaction.
Questions for authors
- Have the authors evaluated the representation intervention on independently generated programs that do not inherit the dynamic-parent selection procedure?
- Does the effect persist in a realistic serialisation where field names collide with values or at higher reasoning effort?
- Could directly estimate run-to-run variance directly, for example with five or more repeated runs of LEGACY and P/Q under one configuration?