acceptodds
Desk rejected at ICLR 2027

Beyond Pass Rates: Attributing Relative-Spatial Failures in Generated Animation Code

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Language models are increasingly utilized to generate animation programs for explanatory videos, yet relative-spatial instructions remain substantially harder than absolute motion or temporal constraints. Existing work establishes this performance gap but largely treats spatial failure as a single phenomenon and evaluates interventions through aggregate pass rates. In this paper, we introduce **RefProbe**, a diagnostic framework that attributes relative-spatial failures to distinct underlying causes using two signals exposed by program generation: (i) the observed placement behavior in the execution trace; (ii) the geometric quantities consulted by the generated program. Our framework assigns failures under a fixed decision procedure and enables interventions to be analyzed by the causes they repair and the behaviors they regress, rather than only by their net effect on pass rate. Across seven open-weight generators and 42,000 generated programs, we find that relative-spatial failures are strongly heterogeneous across both causes and models, with a substantial fraction arising from object-extent errors, reference anchoring failures, and verifier-boundary mismatches. Moreover, interventions with modest aggregate effects often produce large but opposing repair and regression effects at the mechanism level. These results suggest that aggregate pass rates are insufficient for diagnosing spatial generation failures and motivate cause-aware evaluation of program-generating models.

Primary area:
causal reasoning
Keywords:
generate animation programs

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_dgZROpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces REFPROBE, a diagnostic framework to understand why generated animation code fails at relative spatial tasks (e.g., "place object A to the left of object B"). Instead of just looking at overall pass rates, the authors analyze the generated code and its execution trace to assign failures to three specific causes: failure to anchor the target on the referent, ignoring object extent/size, and mismatching the verifier's spatial tolerance convention. They validate this framework using fault injection and counterfactual tests. Evaluating 42,000 programs from seven open-weight models, they find that spatial failures are highly heterogeneous and model-dependent. Furthermore, by testing four prompting interventions, they show that aggregate pass rates hide the fact that interventions often fix some specific errors while simultaneously breaking previously working cases.

Critical strengths
  1. Shifting the focus from "how many" to "why": The paper proposes a concrete, validated way to diagnose spatial failures in code generation. The distinction between ignoring object extent (treating boxes as points) and mismatching the verifier's spatial tolerance convention (failing a strict geometric check but passing a relaxed one) is very useful. It shows that a large chunk of "failures" are actually just boundary cases or slight misunderstandings of the verifier's strictness.

  2. Revealing the "canceling effect" of interventions: The analysis showing that prompts (like telling the model to be more explicit or giving it the oracle predicate) can fix specific errors while breaking previously correct cases is excellent. It exposes a major flaw in how the community currently measures the effectiveness of prompting techniques.

Critical weaknesses
  1. The synthetic nature of the carrier limits generalizability: The model uses a custom, highly simplified 4-function animation runtime. While this allows for perfect diagnostic control, it strips away the messiness of real-world coding (libraries, complex UI frameworks, rendering glitches). The specific causes found here (especially verifier convention mismatches) might be artifacts of this simplified environment.

  2. Little guidance on how to fix the models: The paper diagnoses the problems well, but offers no real solutions. It shows that models fail for different reasons and simple prompting doesn't help much. However, what should researchers actually do to make the models better at spatial reasoning? The paper is purely about evaluation, leaving us with no practical tips on how to fix the underlying issues.

Questions for authors
  1. Real-world transfer: Do you think the three failure modes identified (anchoring, extent, convention) would be equally prominent if you used a more standard library (like p5.js or Three.js) and a vision-based verifier instead of your custom runtime and geometric checks?

  2. Decision order bias: Your decision order classifies based on the first failed predicate. How often does a single generated program contain multiple independent spatial errors (e.g., one anchoring error and one extent error)? Could you report the distribution of the number of distinct causes per failing program?

  3. Mitigation: Given your findings that instruction-level interventions often cause regressions, what is your intuition for a better architecture or training method to solve these issues? Should models be trained to explicitly output their intermediate geometric calculations before writing the final code?

1 comment
AuthorsOpenReview ↗

Response to Reviewer dgZR (Part 1)

W1 / Q1. Generalizability Beyond the Simplified Runtime

We thank the reviewer for raising this important concern. We would like to clarify that REFPROBE employs a controlled animation runtime to isolate spatial failure patterns under well-defined execution semantics, enabling systematic analysis of execution behavior and program provenance.

To investigate the influence of runtime conventions, we changed the coordinate anchoring from center-based to top-left positioning and repeated the baseline evaluation across all seven models, while retaining bounding-box-based verification. The failure distribution changed significantly (χ²(3) = 183.4, p = 1.7 × 10⁻³⁹), with the boundary-band failure share decreasing from 22.2% to 9.7%. This finding is consistent with our observation that 706 of 4,075 baseline separation constraints use corner-based coordinate formulas despite the original center-based API. These results provide direct evidence that runtime conventions influence the observed failure distributions.

More broadly, the three diagnostic patterns concern reference dependencies, object extents, and coordinate conventions, which are also relevant to standard graphics libraries such as p5.js and Three.js. However, their relative prevalence may vary with library-specific semantics and verification methods. Vision-based verification may also introduce additional uncertainty in near-boundary cases. We will clarify these generalizability considerations in the final version and further investigate cross-environment transfer in future work.


W2 / Q3. Practical Implications and Mitigation

We thank the reviewer for raising this important question. Our additional experiments provide practical insights into improving spatial code generation, particularly through more reliable geometric representations, consistent runtime semantics, and verification-guided repair.

First, explicit intermediate reasoning alone does not guarantee correct geometry. Although Probe B requires geometric descriptions before code generation, 530 of 560 generated headers specify only center coordinates rather than object boundaries. An edge-explicit prompt achieves greater repair gains for corner-anchoring failures than for other failures (46.6 versus 19.2 percentage points above the control reference), suggesting the importance of explicitly representing object boundaries and separation margins.

We further evaluated two interventions that directly affect spatial execution performance.

Table R3. Spatial pass rates (%) before and after mitigation interventions.

InterventionBaselineAfter InterventionImprovement
Top-Left Anchor Adjustment59.564.2+4.7 pp
Verification-Gated Intervention59.668.5–72.9+8.9–13.3 pp

Each intervention is compared with its corresponding baseline. Verification-gated results represent an oracle upper bound requiring access to the gold verifier.

As shown in Table R3, aligning coordinate conventions with the runtime improves spatial performance, although the effects vary across models. Selectively applying interventions to failed programs achieves larger improvements under oracle verification, highlighting the potential value of targeted repair over uniform prompting.

Together, these findings motivate a promising direction combining structured geometric planning, API-consistent code generation, and execution-guided correction. We will further discuss these practical implications in the final version.


Q2. Multiple Spatial Failure Patterns

We thank the reviewer for this question. We examined all 1,458 baseline spatial failures to quantify the coexistence of distinct diagnostic patterns. Among these programs, 1,328 (91.1%) exhibit one identifiable pattern, 94 (6.4%) exhibit two distinct patterns, and 36 (2.5%) remain undetermined. Specifically, 1,242 programs fail one relational predicate, while 216 fail two.

Importantly, overlapping behavioral and provenance signals within a single predicate do not necessarily represent independent errors, as they may reflect different aspects of the same incorrect computation. These results indicate that most failing programs exhibit a single identifiable diagnostic pattern, while the coexistence of multiple distinct patterns is relatively uncommon. We will continue to investigate interactions among different failure patterns and their implications for more fine-grained spatial error attribution in future work.

Reviewer_KKr6OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This diagnostic paper explores how LLMs generating animation programs perform well on absolute motion and timing but struggle when one object must be placed relative to another. This gap is known, but its cause is not, and current evaluation relies on aggregate pass rates. The paper introduces REFPROBE, a framework that attributes relative-spatial failures to distinct causes using two views available from generated code: a behaviour view (from the execution trace) and a provenance view (which quantities the program used). Failures are assigned to three causes (missing anchoring on the reference object, ignoring object size, and near-misses within the verifier’s tolerance) plus a residual category, and the procedure is validated with fault injection and counterfactual tests. Across seven open-weight models and 42,000 programs, the authors find that failure causes vary strongly across models. That prompt interventions with small aggregate effects repair some failures while breaking previously correct cases, with the two effects cancelling in the pass rate.

Critical strengths
  1. A diagnostic approach that examines failures in both code and output. By combining a behaviour view with a provenance view, the paper explains failures in a way that pass rates and pixel-level evaluation cannot. It is well positioned against related work: spatial weaknesses are documented in text, multimodal, and video-generation settings, and the paper isolates the setting where models write executable programs, providing a view other approaches do not.
  2. Well-designed experiments with detailed data description and appropriate statistics. The probe set is carefully constructed and described in detail (five categories, 200 items each, a satisfiability guarantee for every item), tolerances are fixed before generation, and the analysis uses appropriate statistical tests (McNemar, Wilson intervals, χ², Cohen’s κ).
Critical weaknesses
  1. Limited generalizability due to model selection. Five of the seven models are Qwen variants, and the selection is not motivated. All models are 2024–2025 releases, although newer open models from other families are available, and no closed models (e.g., GPT, Claude, Gemini) are evaluated, although the motivating prior work used a stronger closed model. Results also differ strongly within the Qwen family (e.g., opposite signs on spatio-temporal items in Figure 2), so conclusions drawn mostly from one family may not generalize to other models or to those used in practice.
  2. An unexplained anomalous result. Olmo-3-7B passes 95.5% of atomic items but only 1.0% of temporal items (Table 1), while writing the correct temporal predicate 94.0% of the time (Table 6). The paper attributes this to composition (§5.2) but does not investigate it. Such an extreme gap suggests a systematic issue (e.g., in how the model uses the animation API) rather than a capability limit, and it raises the question of whether similar artifacts affect other results.
Questions for authors

Questions that could change the assessment:

  1. What motivated the model selection? Could you add non-Qwen models, newer models, and at least one closed model? How do you expect your findings to compare with currently used models?
  2. What causes Olmo-3-7B’s 1.0% temporal pass rate? Could you examine its failing programs?
  3. The results depend on the decision order: checking provenance first relabels 73.2% of failures and raises HrefH_{\text{ref}} from 15.6% to 79.1% (§6.4). How do Table 2 and Table 9 change under this alternative order?
  4. Is the repair-and-break effect specific to the probes, or would any prompt change cause it? A control with simple paraphrases of the baseline instruction would show this, especially since §6.1 notes the probes also act through the form of the prompt. Clarifications:
  5. Were the models run in reasoning or non-reasoning mode? In particular, was Qwen3’s thinking mode enabled?
  6. How was the 12 px relaxed allowance chosen? §5.4 reports sensitivity to halving and doubling the band, but what motivates this specific value?
  7. Why do models in the same family show opposite results in Figure 2? Premature grounding (Table 7a) partly accounts for this, but not for the differences between models.
  8. How do the three contributions, three hypotheses, four failure categories, and two validation checks relate? A short overview, and more concise contributions summarizing the paper, could help.
  9. Figure 1 is very difficult to read when printed; consider splitting or simplifying it.
  10. How does C3 relate to prior work showing that aggregate scores can hide opposing changes, e.g., Yan et al., “Positive-Congruent Training: Towards Regression-Free Model Updates” (CVPR 2021) and Ma et al., “(Why) Is My Prompt Getting Worse? Rethinking Regression Testing for Evolving LLM APIs” (CAIN 2024)? Is the cause-level breakdown the main novelty?
  11. Minor: what hardware was used for serving, and why do DeepSeek-Coder and Olmo not reach 100% executability?
6 comments
AuthorsOpenReview ↗

Response to Reviewer KKr6 (Part 1)

We sincerely thank the reviewer for the thoughtful feedback and valuable suggestions regarding model selection and generalizability.


W1 / Q1. Model Selection and Generalizability

We thank the reviewer for this valuable suggestion. Our original model selection aimed to ensure reproducible comparisons under consistent local vLLM serving and greedy decoding, while covering different model scales and coding specializations. To further assess generalizability, we conducted additional evaluations on three open-weight models and two closed models.

We evaluated Llama-3.1-8B-Instruct, Phi-4, and Qwen2.5-Coder-14B-Instruct under the same protocol of 1,000 tasks and six experimental conditions. Their baseline pass rates are 74.0%, 73.3%, and 92.3%, respectively. The failure distributions remain heterogeneous across models and under both attribution orders. Moreover, all four probes produce both repairs and regressions in these additional models, extending our observations beyond the original model set.

We also evaluated GPT-5.6 Terra and Gemini 4 Argon, achieving baseline pass rates of 99.8% and 99.9%, respectively. These results indicate near-saturated performance on the current benchmark, leaving too few failures for reliable cause-level comparisons. Their intervention responses also differ from those of smaller open-weight models, highlighting the importance of model-specific analysis.

Furthermore, within the evaluated Qwen families, reference-anchoring failures tend to decrease with model scale, from 33.2% to 14.0% for Qwen3-4B/8B and from 9.4% to 1.3% for Qwen2.5-Coder-7B/14B. Although this does not establish a universal scaling trend, it provides additional evidence that failure characteristics vary with model capability.

Overall, these evaluations broaden REFPROBE's empirical coverage and further support its utility for diagnosing heterogeneous failure behaviors across different model families.


W2 / Q2. Investigating the OLMo-3-7B Temporal Anomaly

We thank the reviewer for highlighting this unusual result. To better understand the discrepancy between OLMo-3-7B's temporal predicate accuracy and execution performance, we conducted additional program-level analysis.

Our analysis reveals a consistent temporal scheduling failure in the generated programs. The animation API defaults to start=0, meaning that animations execute simultaneously unless a later start time is explicitly specified. However, OLMo-3-7B assigns a non-zero start in only 3 of 200 temporal programs. In 167 cases, the individual animation actions are correctly generated, but their execution order fails to satisfy the required temporal constraints. When the evaluation conventions are explicitly provided through Probe C, the temporal pass count increases from 2/200 to 64/200.

These findings indicate that the discrepancy primarily arises from the model's failure to translate temporal constraints into executable scheduling instructions, rather than its inability to generate individual animation actions. Importantly, these temporal-only failures are not included in REFPROBE's relative-spatial failure attribution.

To further assess the robustness of our main findings, we separately examined the effects of excluding execution failures and OLMo-3-7B.

Table R1. Regression rates (%) under different robustness settings.

Evaluation SettingProbe AProbe BProbe CProbe O
Original Results37.129.017.313.7
Excluding Crashed Programs35.327.516.913.1
Excluding OLMo-3-7B35.026.616.813.1

As shown in Table R1, the regression rates remain comparable across all four probes under both robustness settings. These results provide additional evidence that our central observation remains consistent: prompting interventions can repair previously failing cases while also introducing regressions. This finding is not driven by the anomalous OLMo-3-7B results.

AuthorsOpenReview ↗

Response to Reviewer KKr6 (Part 2)

Q3. Robustness to Attribution Decision Order

We thank the reviewer for this important question. To further examine the influence of attribution order, we conducted additional analyses using provenance-first attribution and compared the results with the original behavior-first setting.

Table R2 summarizes the changes in the failure distribution.

ModelH_ref (%)H_extent (%)H_conv (%)Unclassified (%)
DS-Coder-6.7B10.8 → 57.030.8 → 0.025.4 → 19.433.0 → 23.7
OLMo-3-7B24.9 → 91.335.1 → 1.916.6 → 1.123.4 → 5.7
Qwen2.5-Coder-7B9.4 → 82.342.2 → 0.528.6 → 10.919.8 → 6.2
Qwen2.5-7B8.7 → 92.861.9 → 1.120.4 → 1.99.1 → 4.2
Qwen2.5-32B-AWQ2.2 → 3.376.1 → 0.018.5 → 18.53.3 → 78.3
Qwen3-4B33.2 → 99.540.9 → 0.020.7 → 0.55.2 → 0.0
Qwen3-8B14.0 → 89.043.0 → 0.033.7 → 6.49.3 → 4.7
Pooled15.6 → 79.144.4 → 0.623.3 → 7.716.8 → 12.6

Table R2. Failure attribution under behavior-first and provenance-first orders. Each entry reports behavior-first → provenance-first.

Although category proportions change with attribution order, the heterogeneity across models remains evident, with the corresponding χ²(18) statistic increasing from 248.3 to 631.8. We also recomputed the repair statistics in Table 9 under provenance-first attribution. For H_ref, the repair rates of probes A/B/C/O are 23.5/26.9/29.0/25.4%, respectively. For H_conv, they are 12.5/41.1/17.0/16.1%. The H_extent category contains only nine failures under this order and is therefore not used for probe-level comparisons.

Importantly, Probe A remains the least effective at repairing H_conv failures under both orders, while Probe B remains the most effective. The corresponding repair rates are 14.5% versus 33.6% under behavior-first attribution and 12.5% versus 41.1% under provenance-first attribution. The regression rates of A/B/C/O also remain unchanged at 37.1/29.0/17.3/13.7%, since they do not depend on failure-category assignments.

These results show that, although attribution order affects category prevalence, the main observations concerning cross-model heterogeneity and opposing repair and regression effects remain supported under both perspectives.


Q4. Prompt Sensitivity and Intervention-Specific Effects

To distinguish the effects of targeted interventions from ordinary prompt variation, we conducted additional controlled experiments across 10 open-weight models and 600 spatial tasks, using identical decoding and verification settings.

The controls include a repeated baseline, two semantically equivalent paraphrases, and a neutral note matched to the additional text introduced by Probe C. The repeated baseline produces byte-identical outputs, confirming the reproducibility of the serving setup. Meanwhile, the two paraphrases change 15.9% and 19.6% of baseline outcomes, and the neutral note changes 14.4%. All four targeted probes repair more previously failing cases than any neutral control. Their regression rates are compared below.

Prompt ConditionRegression Rate
Neutral Controls10.8–18.7%
A. Explicit Rewriting35.4%
B. Grounding Header28.4%
C. Convention Disclosure17.0%
O. Oracle Predicate13.2%

The regression rates of A and B exceed those of all neutral controls, whereas C and O remain within the observed control range. These results indicate that general prompt sensitivity contributes to outcome variation but does not fully account for the observed repair effects. More importantly, they support our central finding that aggregate pass rates can conceal substantial and opposing changes in individual task outcomes.


Q5. Reasoning Mode Configuration

All evaluated models were run in non-reasoning mode. For Qwen3, we explicitly set enable_thinking=False, and none of the 12,000 generated outputs contained thinking tags. For the additional closed models, GPT-5.6 Terra used reasoning effort set to none, while Gemini 4 Argon used a thinking budget of 0. These settings ensure that the reported comparisons use non-reasoning inference configurations.


Q6. Selection of the 12 px Relaxed Allowance

We thank the reviewer for asking about the tolerance setting. The 12 px allowance was specified before generation as a fixed, small overlap tolerance for separation constraints, rather than being fitted to the evaluation results. We additionally examined the sensitivity of failure attribution to nearby thresholds. Among the 1,458 relative-spatial failures, the proportion of separation near misses within the allowance is 21.1% at 10 px and 21.7% at 12 px. These results show that the relevant attribution statistics remain stable under nearby tolerance settings, supporting the robustness of our analysis to the precise threshold choice.

AuthorsOpenReview ↗

Response to Reviewer KKr6 (Part 3)

Q7. Understanding Within-Family Differences in Figure 2

We thank the reviewer for this insightful question. Our additional analysis suggests that the within-family differences in Figure 2 are partly explained by variations in how models represent spatial relations.

Specifically, some models frequently generate concrete displacement commands (move_by) instead of relational predicates. Among the 200 spatio-temporal tasks, Qwen2.5-Coder-7B produces 124 such displacements, compared with 19 for Qwen2.5-7B. Similarly, Qwen3-4B produces 124, compared with 28 for Qwen3-8B. These differences closely correspond to their respective comprehension-generation gaps of −58.0 versus +12.0 and −53.0 versus −1.0.

Importantly, these displacement expressions are not necessarily incorrect. For example, 52 of 59 displacements generated by Qwen2.5-Coder-7B on spatio-temporal canvas-edge tasks satisfy the gold predicates. This provides concrete evidence that the observed gaps partly reflect differences in output representation and evaluation sensitivity, rather than differences in spatial reasoning ability alone. We will further clarify this distinction and include a complementary predicate evaluation that recognizes geometrically equivalent displacement expressions.


Q8. Relationship Between Contributions, Hypotheses, and Validation

We appreciate the reviewer's suggestion to clarify the organization of REFPROBE. The three hypotheses (HrefH_{ref}, HextentH_{extent}, and HconvH_{conv}), together with the residual category, define four diagnostic outcomes under the specified attribution order. Fault injection (E1) and counterfactual provenance testing (E5) provide complementary evidence for evaluating these diagnostic assignments.

These components directly support our three contributions. C1 (§3) introduces the diagnostic framework by combining execution behavior and program provenance. C2 (§5, E1–E5) characterizes failure patterns across 42,000 generated programs from seven models and evaluates the diagnostic procedure. C3 (§6, E6–E9) examines how four prompting interventions repair different failure patterns while also introducing regressions in previously successful programs.


Q9. Readability of Figure 1

We thank the reviewer for this helpful suggestion. In the final version, we will improve the readability of Figure 1 by retaining the core diagnostic pipeline, including the failing example, execution behavior, program provenance, and attribution procedure, in the main text. The additional validation and intervention components will be moved to the appendix.


Q10. Distinction from Prior Regression Analysis

We thank the reviewer for pointing out these relevant studies. We agree that previous work, including Yan et al. (2021) and Ma et al. (2024), has established the importance of examining both improvements and regressions beyond aggregate performance metrics. We would like to clarify the additional perspective offered by REFPROBE.

Our work extends this line of analysis by examining outcome changes at the level of relative-spatial failure patterns and their underlying program computations. Specifically, REFPROBE combines execution behavior with program provenance to identify which failure patterns are repaired by different prompting interventions and to investigate how previously successful programs become incorrect. For example, the signed-slack analysis in §6.3 provides geometric evidence of how explicit reference-frame rewriting can introduce regressions. Therefore, while prior studies establish the importance of evaluating opposing performance changes, REFPROBE provides a complementary diagnostic perspective that connects these changes to interpretable spatial failure patterns and program-level evidence.


Q11. Serving Hardware and Executability

We thank the reviewer for requesting these implementation details. All model generations were served using vLLM on a shared multi-GPU NVIDIA H100 server, with one GPU assigned per model. Program execution, verification, and provenance analysis were performed on CPUs.

Regarding executability, DeepSeek-Coder-6.7B and OLMo-3-7B produced 3 and 4 non-executable programs out of 1,000 baseline generations, respectively. Our inspection indicates that these failures resulted from issues in the generated code, including calls to APIs or references to objects not provided by the task environment, undefined variables, variable redeclarations, and syntax errors. These findings explain the small deviations from 100% executability. We will provide executability statistics for all models and experimental conditions in the final version.

Reviewer_qckXOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies relative-position errors in animation code generated by LLMs. For example, when asked to “place the square to the left of the circle,” the code may run, but the final position may not follow the instruction. The authors propose RefProbe, which checks the positions after the code runs and what object information was used to calculate them. It groups failures into four categories:

  • Reference errors: The position calculation does not use the specified reference object’s position or size.
  • Object-size errors: The code compares only the objects’ centers and ignores their sizes, causing them to overlap.
  • Evaluation-rule differences: The position is close to correct but does not meet the evaluator’s threshold.
  • Other unclassified errors.

The authors test seven models on 1,000 tasks in a simplified JavaScript animation environment. Each model is tested under six settings, producing 42,000 outputs in total. One setting generates only formal constraints, while the others generate animation code. They also test four ways to change the prompt, such as clearly naming the edges to compare or directly providing the correct formal constraints. The main finding is that a prompt change can fix some cases that previously failed while breaking others that previously passed. The overall pass rate alone therefore does not show the full effect.

Critical strengths
  1. The research question is clear, and the diagnostic approach is useful: The paper examines not only how often models succeed, but also where they fail and which cases improve or get worse after a prompt change. This provides more specific information than the overall pass rate.

  2. Combining code analysis with execution results is a reasonable approach: The same position error can come from different code. Checking whether the code uses information about the reference object reveals differences that cannot be seen in the final output alone.

Critical weaknesses
  1. The framework currently classifies error patterns rather than clearly identifying their causes. Passing when objects are treated as points only shows that the result meets a center-based comparison. It does not prove that ignoring object size caused the error. Similarly, passing under a relaxed threshold does not mean the model used a different spatial rule; it may simply have made a small calculation error.

Fault-injection tests show that the tool can detect predefined errors, and counterfactual tests support its checks for reference dependence. However, these tests do not independently show that all three categories capture the actual causes of generation failures.

  1. The reported error distribution depends heavily on the chosen classification order. The authors check threshold issues first, size issues second, and reference issues last. However, Section 6.4 shows that checking reference information first would reclassify 73.2% of failures and increase the share of reference errors from 15.6% to 79.1%.

The authors acknowledge that these orders answer different questions. Still, this means the finding about “which cause is most common” is not stable. Since causes can occur together, allowing multiple labels or reporting the two analysis dimensions directly would make more sense than assigning one label to each failure.

Questions for authors
  1. How do the authors distinguish between an error that matches a certain rule and an error actually caused by that rule?

  2. Since changing the decision order reclassifies 73.2% of cases, why do the main results assign only one label per failure, rather than allow multiple labels or jointly analyze placement behavior and reference dependence?

  3. Could the improvements and regressions simply reflect general sensitivity to prompt changes?

3 comments
AuthorsOpenReview ↗

Response to Reviewer qckX (Part 1)

We thank the reviewer for the constructive feedback and for recognizing the value of REFPROBE's diagnostic approach.


W1 / Q1. Clarification and Additional Evidence for Failure Attribution

We sincerely thank the reviewer for raising this important concern. We would like to provide further clarification of REFPROBE's attribution framework and additional empirical evidence to better address the distinction between observed failure patterns and their underlying mechanisms. REFPROBE combines execution behavior with program-level provenance, rather than relying solely on observed placement outcomes. To further substantiate this approach, we conducted three complementary analyses:

  1. Mechanism identification. We identified a systematic coordinate-anchor convention mismatch: although the runtime defines x and y as object-center coordinates, 706 of 4,075 separation constraints use formulas consistent with top-left anchoring. For 320 such formulas without additional margins, geometric analysis correctly predicts all 191 passing and 129 failing outcomes.

  2. Targeted intervention. Changing the runtime anchor convention to top-left reduces corner-anchor-related failures from 187 to 19 in a separate intervention run, providing direct evidence that the coordinate convention contributes to these failures.

  3. Counterfactual and provenance analysis. When the referent is displaced, 226 of 227 programs attributed to HrefH_{ref} fail to follow it, supporting the reference-dependence diagnosis. Further inspection of 647 HextentH_{extent} cases reveals heterogeneous computational behaviors, motivating a more precise characterization of this category.


W2 / Q2. Decision-Order Sensitivity and Joint Attribution

We thank the reviewer for this valuable suggestion. We would like to provide further clarification regarding the design of REFPROBE's attribution framework. The single-label attribution provides a mutually exclusive partition of failures, ensuring that each case is counted only once when analyzing repairs and regressions.

The 73.2% reclassification reflects the different perspectives of behavior-first and provenance-first attribution. The former focuses on how the generated placement violates spatial constraints, while the latter examines whether the computation depends on the referent's geometry. These perspectives provide complementary information about the same failure.

To further examine this issue, we additionally analyzed the joint Behavior × Provenance distributions and compared results under both attribution orders. Our key findings remain consistent across both perspectives: failure distributions differ across models, all prompting interventions produce both repairs and regressions, and explicit reference-frame rewriting remains the least effective intervention for repairing boundary-band failures. Thus, while category prevalence depends on the attribution order, the central findings of REFPROBE are supported by both perspectives. The joint analysis complements the original single-label framework by revealing overlapping failure characteristics without sacrificing the interpretability of repair and regression statistics.


Q3. Distinguishing Targeted Interventions from General Prompt Sensitivity

To assess whether the observed repairs and regressions can be explained by general prompt sensitivity, we conducted additional controlled experiments using two semantically equivalent paraphrases and a neutral-note control, evaluated alongside the four original probes across 10 models and 600 spatial tasks. The neutral controls change 14–20% of task outcomes, establishing a reference for ordinary prompt sensitivity. Importantly, all four targeted probes repair a larger proportion of previously failing cases than any neutral control. We further compare their regression rates, defined as the proportion of previously successful cases that become failures:

Prompt ConditionRegression Rate
Neutral Controls10.8–18.7%
A. Explicit Rewriting35.4%
B. Grounding Header28.4%
C. Convention Disclosure17.0%
O. Oracle Predicate13.2%

The regression rates of A and B exceed those of all neutral controls, whereas C and O remain within the observed control range. These results suggest that general prompt sensitivity contributes to outcome variation but does not fully explain the observed repair effects. More importantly, the simultaneous occurrence of repairs and regressions supports our central finding that aggregate pass rates alone are insufficient to characterize how interventions affect model behavior.

AuthorsOpenReview ↗

Hugely appreciate the suggestions and looking forward to engaging further!

Thank you again for your constructive feedback. We have responded to your concerns point by point and hope our responses help clarify the issues you raised. If you have any further questions, we would be happy to provide additional clarification. Thank you again for your time and consideration!

AuthorsOpenReview ↗

Follow-up on Our Rebuttal and Request for Reassessment

Dear Reviewer,

We would appreciate your feedback on our rebuttal when you have an opportunity. Please let us know if any concerns remain or further clarification would be helpful.

If our responses address your concerns, we would be grateful if you could reconsider your assessment and score.

Thank you for your time and consideration.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Rebuttal and Additional Evidence

Dear Area Chair and Reviewers,

We sincerely thank all reviewers for their constructive feedback. As the discussion period closes, we summarize the strengths recognized by reviewers and the additional evidence provided in our rebuttal.

Strengths recognized by the reviewers.

  • Problem formulation. REFPROBE shifts relative-spatial code generation evaluation from aggregate pass rates to diagnosing failure patterns and intervention effects (qckX, KKr6, dgZR).
  • Diagnostic framework. Combining execution behavior with program provenance reveals spatial failure patterns that final outputs alone cannot explain (qckX, KKr6, dgZR).
  • Experiments. The evaluation covers seven open-weight models and 42,000 programs, with carefully designed tasks and statistical analyses (KKr6).
  • Intervention analysis. The framework reveals how prompting interventions simultaneously repair failures and introduce regressions, effects concealed by aggregate pass rates (qckX, KKr6, dgZR).

Reviewer qckX. Our rebuttal provides additional evidence addressing the reviewer's concerns:

  • Failure attribution (W1, Q1). We identified 706 of 4,075 separation constraints using top-left formulas despite the center-based runtime. Adjusting the anchor convention reduces related failures from 187 to 19. Counterfactual tests also confirm reference-dependence violations in 226 of 227 attributed programs.
  • Decision-order sensitivity (W2, Q2). Joint Behavior × Provenance analysis shows that, despite substantial changes in category proportions, cross-model heterogeneity and opposing repair-regression effects persist under both attribution orders.
  • Prompt sensitivity (Q3). Across 10 models and 600 tasks, all four targeted probes repair more failures than neutral controls. Probes A and B exhibit regression rates of 35.4% and 28.4%, exceeding the neutral-control range of 10.8–18.7%.

Reviewer KKr6. We extended the empirical evaluation and addressed the reviewer's specific questions:

  • Generalizability (W1, Q1). Three additional open-weight models confirm heterogeneous failure patterns and opposing repair-regression effects. Two closed models achieve near-saturated baseline pass rates (99.8% and 99.9%), limiting cause-level comparisons.
  • OLMo temporal anomaly (W2, Q2). Only 3 of 200 temporal programs specify a nonzero start time. Explicit runtime conventions improve temporal passes from 2/200 to 64/200. Excluding OLMo or crashed programs leaves the main findings broadly unchanged.
  • Attribution robustness (Q3). Under provenance-first attribution, reference-dependence failures increase from 15.6% to 79.1%, but cross-model heterogeneity persists. Probe B remains more effective than Probe A at repairing boundary-band failures under both orders.
  • Additional clarifications (Q5–Q11). We clarified reasoning configurations, tolerance selection, within-family differences, framework components, implementation details, and distinctions from prior regression studies.

Reviewer dgZR recognized the diagnostic value of REFPROBE and its analysis of opposing repair-regression effects, assigning a rating of 3 (Weak Acceptance). We further addressed the reviewer's concerns:

  • Runtime generalizability (W1, Q1). Switching to top-left anchoring reduces boundary-band failures from 22.2% to 9.7%, demonstrating the influence of runtime conventions while clarifying cross-environment limitations.
  • Overlapping failures (Q2). Among 1,458 spatial failures, 91.1% exhibit one identifiable pattern, 6.4% exhibit two, and 2.5% remain undetermined.
  • Practical mitigation (W2, Q3). Coordinate-convention alignment improves spatial pass rates by 4.7 percentage points, while oracle verification-gated interventions yield gains of 8.9–13.3 points, highlighting the potential of targeted repair.

Together, these additional experiments strengthen REFPROBE's central findings: spatial failures exhibit heterogeneous patterns across models, and aggregate pass rates conceal opposing repair and regression effects. The extended evaluations and robustness analyses further clarify the framework's applicability and limitations.

We thank the reviewers and Area Chair for their time and consideration, and hope our responses and additional evidence will inform the final assessment.

Best regards,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.