- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces REFPROBE, a diagnostic framework to understand why generated animation code fails at relative spatial tasks (e.g., "place object A to the left of object B"). Instead of just looking at overall pass rates, the authors analyze the generated code and its execution trace to assign failures to three specific causes: failure to anchor the target on the referent, ignoring object extent/size, and mismatching the verifier's spatial tolerance convention. They validate this framework using fault injection and counterfactual tests. Evaluating 42,000 programs from seven open-weight models, they find that spatial failures are highly heterogeneous and model-dependent. Furthermore, by testing four prompting interventions, they show that aggregate pass rates hide the fact that interventions often fix some specific errors while simultaneously breaking previously working cases.
Critical strengths
-
Shifting the focus from "how many" to "why": The paper proposes a concrete, validated way to diagnose spatial failures in code generation. The distinction between ignoring object extent (treating boxes as points) and mismatching the verifier's spatial tolerance convention (failing a strict geometric check but passing a relaxed one) is very useful. It shows that a large chunk of "failures" are actually just boundary cases or slight misunderstandings of the verifier's strictness.
-
Revealing the "canceling effect" of interventions: The analysis showing that prompts (like telling the model to be more explicit or giving it the oracle predicate) can fix specific errors while breaking previously correct cases is excellent. It exposes a major flaw in how the community currently measures the effectiveness of prompting techniques.
Critical weaknesses
-
The synthetic nature of the carrier limits generalizability: The model uses a custom, highly simplified 4-function animation runtime. While this allows for perfect diagnostic control, it strips away the messiness of real-world coding (libraries, complex UI frameworks, rendering glitches). The specific causes found here (especially verifier convention mismatches) might be artifacts of this simplified environment.
-
Little guidance on how to fix the models: The paper diagnoses the problems well, but offers no real solutions. It shows that models fail for different reasons and simple prompting doesn't help much. However, what should researchers actually do to make the models better at spatial reasoning? The paper is purely about evaluation, leaving us with no practical tips on how to fix the underlying issues.
Questions for authors
-
Real-world transfer: Do you think the three failure modes identified (anchoring, extent, convention) would be equally prominent if you used a more standard library (like p5.js or Three.js) and a vision-based verifier instead of your custom runtime and geometric checks?
-
Decision order bias: Your decision order classifies based on the first failed predicate. How often does a single generated program contain multiple independent spatial errors (e.g., one anchoring error and one extent error)? Could you report the distribution of the number of distinct causes per failing program?
-
Mitigation: Given your findings that instruction-level interventions often cause regressions, what is your intuition for a better architecture or training method to solve these issues? Should models be trained to explicitly output their intermediate geometric calculations before writing the final code?
Response to Reviewer dgZR (Part 1)
W1 / Q1. Generalizability Beyond the Simplified Runtime
We thank the reviewer for raising this important concern. We would like to clarify that REFPROBE employs a controlled animation runtime to isolate spatial failure patterns under well-defined execution semantics, enabling systematic analysis of execution behavior and program provenance.
To investigate the influence of runtime conventions, we changed the coordinate anchoring from center-based to top-left positioning and repeated the baseline evaluation across all seven models, while retaining bounding-box-based verification. The failure distribution changed significantly (χ²(3) = 183.4, p = 1.7 × 10⁻³⁹), with the boundary-band failure share decreasing from 22.2% to 9.7%. This finding is consistent with our observation that 706 of 4,075 baseline separation constraints use corner-based coordinate formulas despite the original center-based API. These results provide direct evidence that runtime conventions influence the observed failure distributions.
More broadly, the three diagnostic patterns concern reference dependencies, object extents, and coordinate conventions, which are also relevant to standard graphics libraries such as p5.js and Three.js. However, their relative prevalence may vary with library-specific semantics and verification methods. Vision-based verification may also introduce additional uncertainty in near-boundary cases. We will clarify these generalizability considerations in the final version and further investigate cross-environment transfer in future work.
W2 / Q3. Practical Implications and Mitigation
We thank the reviewer for raising this important question. Our additional experiments provide practical insights into improving spatial code generation, particularly through more reliable geometric representations, consistent runtime semantics, and verification-guided repair.
First, explicit intermediate reasoning alone does not guarantee correct geometry. Although Probe B requires geometric descriptions before code generation, 530 of 560 generated headers specify only center coordinates rather than object boundaries. An edge-explicit prompt achieves greater repair gains for corner-anchoring failures than for other failures (46.6 versus 19.2 percentage points above the control reference), suggesting the importance of explicitly representing object boundaries and separation margins.
We further evaluated two interventions that directly affect spatial execution performance.
Table R3. Spatial pass rates (%) before and after mitigation interventions.
Each intervention is compared with its corresponding baseline. Verification-gated results represent an oracle upper bound requiring access to the gold verifier.
As shown in Table R3, aligning coordinate conventions with the runtime improves spatial performance, although the effects vary across models. Selectively applying interventions to failed programs achieves larger improvements under oracle verification, highlighting the potential value of targeted repair over uniform prompting.
Together, these findings motivate a promising direction combining structured geometric planning, API-consistent code generation, and execution-guided correction. We will further discuss these practical implications in the final version.
Q2. Multiple Spatial Failure Patterns
We thank the reviewer for this question. We examined all 1,458 baseline spatial failures to quantify the coexistence of distinct diagnostic patterns. Among these programs, 1,328 (91.1%) exhibit one identifiable pattern, 94 (6.4%) exhibit two distinct patterns, and 36 (2.5%) remain undetermined. Specifically, 1,242 programs fail one relational predicate, while 216 fail two.
Importantly, overlapping behavioral and provenance signals within a single predicate do not necessarily represent independent errors, as they may reflect different aspects of the same incorrect computation. These results indicate that most failing programs exhibit a single identifiable diagnostic pattern, while the coexistence of multiple distinct patterns is relatively uncommon. We will continue to investigate interactions among different failure patterns and their implications for more fine-grained spatial error attribution in future work.