- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper investigates model repair in compositional reasoning tasks. It observes that updating a model to correct a prediction for a combined change can inadvertently break previously correct predictions for its constituent changes, a phenomenon the paper characterizes as error relocation. The main contribution is an instance-level evaluation framework that distinguishes constituent-preserving repair from error relocation, capturing behaviors that endpoint accuracy alone can conceal. The framework is evaluated primarily on a synthetic 2D segment-intersection task using both coordinate and rendered-image inputs. The experiments show that constituent regression can occur frequently even when an update improves joint correctness overall, and further examine how representation, change supervision, and regression-control objectives affect this repair–preservation trade-off.
Critical strengths
- An intuitive distinction between endpoint correction and constituent-preserving repair.
The paper makes a neat observation: fixing an endpoint error does not necessarily mean the model preserves the correct intermediate parts. Tracking this at the instance level separates true repair from mere error relocation, something aggregate accuracy simply misses. I found this evaluation angle intuitive and clean.
- Well-crafted figures.
The figures are clean, informative, and visually well put together, making the setup and key takeaways easy to grasp at a glance.
Critical weaknesses
- Mismatch between the broad claims and the narrow empirical scope.
To be clear, presentation issues alone would not lead me to reject the paper. My main concern is the tension between the broad claims and the narrow evidence. The paper frames repair versus relocation as a general evaluation principle for model updates, but virtually all evidence comes from a synthetic segment-intersection toy problem.
Even the image-based experiments simply re-render this same geometric task. The authors concede that "all tasks concern controlled geometry" and acknowledge that cross-renderer and cross-semantic-task generalization remains unestablished. For an evaluation framework, demonstrating applicability across different model families or realistic domains is essential. Without evidence beyond this toy setting, it is difficult to tell whether this phenomenon matters broadly or is an artifact of the setup.
- The presentation is overly defensive and fragmented, burying the main story.
Instead of explaining why the central finding matters, the paper gets bogged down in secondary controls and qualifications. The writing feels less like a confident scientific paper and more like hedging against every critique. Almost every result is immediately undercut by a disclaimer:
- "This analysis did not establish…"
- "Residual imbalance … limit its scope."
In the alternative explanations section, for example, the text introduces matched endpoint difficulty only to immediately walk back its interpretation; the next paragraph repeats this by stating that same-instance association cannot prove a mechanism. This reads like a disjointed series of disclaimers rather than a clear narrative. Moving these checks to the appendix would allow the main text to focus on the core evaluation idea.
Questions for authors
-
Core takeaway: What is the main claim you want the reader to take away? Is it simply that constituent regression can happen, that it happens routinely under common fine-tuning methods, or that these metrics should become standard practice for model updates? Clarifying this would help calibrate expectations for the experimental section.
-
Narrative flow: The idea is good, but the story gets lost in the writing. I would suggest looking at accepted papers, say, at ICLR, to see how they structure their narrative—especially in the introduction and experiments, where the problem is framed and the central finding is made clear right away. With that in mind, how would you rewrite the introduction to better convey this story and highlight the central finding?