acceptodds
Under review as a conference paper at ICLR 2027

Same Task, Different Representation: When LLM Success Fails to Transfer

Abstract

Large language models (LLMs) often solve tasks using familiar output representations, such as implementing Boolean logic in Verilog/Python or expressing chess moves in standard notation. Yet users may need the same task solved in a different representation, requiring models to apply acquired skills under new constraints. Does demonstrated success transfer when the required output representation changes, and can additional reasoning recover lost success? We thus introduce Paired Representation Robustness (PRR), which measures the fraction of problems a model already solves that remain correct under an alternate output representation. The semantic task stays fixed, and the alternate representation’s rules are explicitly supplied. We evaluate five LLMs on 500 Boolean logic problems (Python code → Minecraft Redstone circuits) and 1,000 chess puzzles (standard move notation → a synthetic cipher notation, Syn-Chess). With CoT, and additional reasoning disabled, GPT-5.6-Sol solves 98.8% of logic problems in Python but retains only 1.01% of those successes in Redstone; it identifies the correct first move in 86.6% of chess puzzles but retains only 0.69% under Syn-Chess. On a held-out subset of 100 Syn-Chess puzzles, adaptive reasoning raises first-move PRR from below 4% to approximately 79% for both GPT-5.6-Sol and GPT-5.6-Luna, at higher inference cost. These findings reveal a barrier to applying demonstrated skills under new output requirements: transfer can collapse even with explicit CoT prompting, while adaptive reasoning recovers much of the lost success at over 10× the baseline inference cost. Output representation thus emerges as a consequential factor in both the reliability and computational cost of LLM problem solving.

Primary area:
datasets and benchmarks
Keywords:
Large language models, benchmark evaluation, evaluation robustness, representation invariance, model robustness

Reviews

Among:
Shape:
1234this paper, Bottom 9%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.50 over 4 reviews, above 9% of the others.
Reviewer_mVjaOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper proposed an interesting analysis between so called ``same-task'' (bool -> Minecraft, UCI -> syn-chess) has difficult representation transfer. It claimed that the transfer fails from the two experiments and need more than 10 times cost reasoning to recover the gain.

Critical strengths
  1. The topic is interesting and the two case study are interesting
  2. The experiments are validated via baseline comparison and human annotation
Critical weaknesses

I have significant concerns about the basis task definition of the paper so before this has been clarified, I am hard to accept the following experiments and claims:

Most importantly, the paper claims that you have same task but essentially the tasks are not the same, and even these are quite different. Boolean task is just a combination task but while you want to create items in minecraft, you need to know basic knowledge in minecraft. It is true that you may give some rules but there are a lot of reasoning behind for example you need to understand the materials, goods, and even what means the combination in minecraft! The same is true for the the second case: - syn-chess needs understand the rules, calculating the moving distance, which in the prompt you banned this skills. I could only admit that there are indeed some skills are shared but definitely not the same tasks. Therefore, the two case you found would not support the claim ``LLM representation does not transfer'' as they might transfer, and the failure is due to the task they perform needing more skills.

So you may need to test a more controllable application pair such as - you need to compare boolean representation with a de-gamified grid-circuit representation - something similar.

Despite all above,

  1. You force reasoning less than 40 words, and the RQ2 allows that, which gives a results from 3% to 79%. It is a sign that the task needs explicit reasoning but not difficulty of transfer
  2. You are trying to argue against the general transfer difficulty but only two fields are given. I understand exhaustively running through all cases is impossible but I would suggest select one field - for example game? health? math? and narrow down your claim
  3. Only 100 tests are given for RQ2
  4. It could be better to explain Claude Opus 4.8 has 66.71% on Syn-Chess but GPT is less than 2%.
  5. You need to give demographics for human participants and their knowledge on chess and minecraft.

Given the above concerns, I do not think the paper could be revised during this round. I therefore recommend rejection.

Questions for authors

See above please.

4 comments
AuthorsOpenReview ↗

Response to Reviewer mVja

We thank the reviewer for their thorough review and for recognizing the value of the two case studies and their validation.

W1: Same Task, Different Representation

See common response C2, which clarifies task preservation and separates solving, encoding, and coordinating the two. Python and Redstone must implement the same Boolean function; UCI and Syn-Chess must solve the same position. We do not claim that the two representations are equally difficult. The additional skills raised by the reviewer are the construction and coordination requirements explicitly discussed in Section 3.3. PRR asks whether demonstrated native success survives these output requirements.

Regarding "calculating the moving distance," the prompt does not ban this skill or the use of ordinary chess knowledge. Standard chess rules and board states remain unchanged; only the final notation changes.

Redstone is already presented through an ARC-AGI-inspired symbolic matrix, with component behavior and placement examples supplied, rather than interactive gameplay. A fully de-gamified grid-circuit comparison would be a useful further control; we do not equate the existing matrix interface with that additional experiment.

W2: Rule Comprehension and Human Study

See common response C2. Given the UCI move, moving side, and physical piece, Sol correctly encodes 55/55 moves into Syn-Chess in one request. Sol & Luna also succeed on 10/10 & 9/10 Redstone comprehension probes, including turning an existing wire toward a lamp and branching it to a second lamp while preserving the original connection. These test wire continuity and spatial layout, not just component names, supporting comprehension of the minecraft supplied rules.

Appendix E provides a complementary human check: two participants achieve 100% and 98% first-move accuracy on 50 puzzles each, given a tutorial similar to the information given in the LLM prompts, showing that the supplied rules can be applied while solving.

ParticipantFirst-move accuracyFull-puzzle accuracy
1100% (50/50)94% (47/50)
298% (49/50)92% (46/50)

Using experienced players and low-rated puzzles reduces the influence of native chess difficulty without first revealing the solutions. We will clarify participant-background and protocol details; this is a small usability study, not a population-level baseline.

W3: Word Limit and Additional Reasoning

See common response C3. On 500 puzzles, removing the 40-word limit leaves PRR below 1% for both GPT models. Removing the limit and placing reasoning before the answer improves transfer, but Sol still loses 64.39% and Luna 93.14% of their native successes. Thus, the word limit alone does not explain the collapse. We will include these controls in our final draft.

RQ2 retains the same visible 40-word restriction and changes API reasoning effort, so its recovery is not obtained by lifting that restriction. Recovery with additional compute addresses the paper's question, rather than contradicting it: how much native success can be recovered, and at what cost? (common response C4)

W4: Scope and Sample Size

We thank the reviewer for this suggestion and will include an in-depth analysis of Claude's successes and failures in the final draft to better understand the Claude-GPT performance gap. We agree that two domains do not support a universal transfer and will narrow down our claims.

For RQ2, we sampled one puzzle from each consecutive block of ten after sorting the 1,000 puzzles by rating, ensuring coverage across the benchmark's difficulty range. We will clarify this sampling procedure in the paper.

If these clarifications and additional results address the concerns to the reviewer's satisfaction, we respectfully ask that they consider raising their evaluation score.

Reviewer_mVjaOpenReview ↗

Response

Thanks the author response. It is nice to hear the authors are willing to do additional experiments for human annotation and my concerns for W3 and W4.

For W1, it is still not convincing - that the two tasks are same - so if you mean the sub-tasks are the same, the overall contribution should be narrowed for description and then in which sense the contribution will be limited. And i am confused by We do not claim that the two representations are equally difficult. .. the meaning of difficult is no clear.

In conclusion - all the concerns are not resolved so due to the limited time, I maintain my score and suggest the paper to be resubmitted :).

AuthorsOpenReview ↗

Follow-up Response to Reviewer mVja

We thank the reviewer for their quick follow-up and the opportunity to clarify W1. Below we clarify the definitions and the scope of our work as per your request.

Definition Clarifications

Below we share an example to explain what we mean by "same task"

In chess domain:

Consider one fixed chess position where an accepted first move takes White's rook from c1 to c8. UCI writes c1c8; Syn-Chess writes Qc1 B 7. Both specify the same move on the same board. The model sees the same board in both conditions and must solve it each time. Both answers (the model’s first move) are converted back into a board move and checked against the same list of correct moves. Thus, only the way the answer is written differs.

By "difficulty," we mean the additional work of expressing a solution: here, calculating displacement and applying the cipher. We do not mean a different mating objective or identical computation in the two conditions. This is consistent with the distinction between informational and computational equivalence in Larkin and Simon (1987), Why a Diagram Is (Sometimes) Worth Ten Thousand Words: equivalent information need not support equally efficient processing.

Finally, our contribution is not to establish that these extra operations exist, but to measure how much demonstrated success survives each output requirement. The new controls make this distinction concrete. On the same 65 native-first-move-correct Luna instances within the 100-puzzle sample, with none reasoning and a 4K allowance:

Output requirementRetained native first-move successes
Ciphered piece tag, unchanged UCI coordinates26/65 (40.00%)
Displacement-only notation3/65 (4.62%)
Full Syn-Chess4/65 (6.15%)

Retention remains low in both conditions requiring displacement output. These results quantify which output requirements retain native success; they do not establish that the model's underlying chess skill disappeared.

In Redstone domain:

the specified Boolean function and its truth table remain fixed, but producing the output additionally requires component knowledge, circuit synthesis, and spatial layout.

Scope Clarifications

  • We will distinguish Redstone's less-controlled implementation-transfer setting from the chess notation change, rather than present both as pure symbol recoding.
  • We will replace unqualified "same task" with "same semantic task" in the title and claims. The contribution will be stated as retained correctness under changed output requirements on instances the model already solves, not equivalence of complete computational tasks or a general inability of LLMs to transfer.

We hope this example and the clarified scope address the reviewer's remaining concern.

Reviewer_mVjaOpenReview ↗

Response

I thank the authors efforts.

W1. This is still the same sub-tasks if I may ... and the example does not fully resolved my concerns as in this example, the latter one contains additional movement notation. In another example, the redstone has game knowledge which is overlooked by this response.

The rest W3,W4 have not been resolved. Only when the results could be made, and we see the example, we could get the chance to see whether it has been resolved.

Therefore, I maintain my score. I hope this active discussion has explain my position and I will leave it to AC for their decision.

Reviewer_MzbqOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates whether LLMs can retain their problem-solving abilities when the required output representation changes. It introduces a benchmark, Paired Representation Robustness (PRR), which measures the percentage of correctly solved problems under a different output requirement. Authors evaluated two tasks: Boolean logic (Python setting vs. Minecraft Redstone setting) and chess (standard UCI notation vs. Syn-Chess notation) and showed that performance degraded in the alternative format, but performance could be recovered using additional inference-time reasoning.

Critical strengths
  • The chess task is valid and shows that LLMs could perform worse in unfamiliar chess notations.
  • Useful error analysis looking into the output format failures and reasoning failures.
Critical weaknesses
  • My main concern is the incremental contribution. Many studies in 2023/2024 have already looked into model sensitivity to formats and task conventions. It is unclear why we need a benchmark with new output requirements if the performance could be easily recovered from more reasoning efforts. It is also unclear if few-shot prompting or SFT could easily recover the performance if the models are just unfamiliar with the output requirements.
  • Although both the python programs and the Minecraft Redstone task require the same boolen logics, the two tasks are not equivalent, as what the paper has claimed. Redstone requires much more understanding in the problem settings. The performance drop could be due to the models not comprehending Redstone game rules.
  • The tasks are rather limited. Only puzzle/game kind of tasks are considered.
Questions for authors
  • how is the human baseline?
  • why is the reasoning effort restricted to at most two short sentences?
2 comments
AuthorsOpenReview ↗

Response to Reviewer Mzbq

We thank the reviewer for their thorough review and for recognizing the validity of the chess setting and the usefulness of the error analysis.

W1: Contribution and Recovery

See common response C1 for the motivation. Section 2 discusses prior format-sensitivity studies, including FormatSpread, Reasoning or Reciting?, and GSM-Plus. Our contribution is not simply that formats affect accuracy: PRR measures the loss of demonstrated success on the exact problems the same model already solves, with the semantic target fixed and alternate rules supplied.

Recovery with additional reasoning answers our second question: how much native success can be recovered, and at what cost (common response C4). It does not establish that native success transfers reliably at the base setting.

Few-shot prompting. We add three solved examples to both native and alternate chess prompts and observe that few-shot prompting does not help recover Syn-Chess retention, while the Native accuracy improves slightly, in this experiment.

ModelPromptNative (Step 2)PRR (Step 3)
GPT-5.6-LunaOriginal56.00% (56/100)1.79% (1/56)
GPT-5.6-LunaThree-shot59.00% (59/100)1.69% (1/59)

We select 100 puzzles by randomly sampling one from each consecutive block of ten, from our full benchmark, to preserve the difficulty coverage.

Cost-matched repeated sampling likewise fails to recover the router's retained successes (common response C4; our reply to Reviewer 7TwY, W3). Thus, the tested in-context examples and repeated attempts do not recover retention comparable to the reasoning-enabled policies.

W2: Task Preservation and Rule Comprehension

See common response C2. Python and Redstone must implement the same Boolean function and match the same truth table. We do not claim that constructing the two outputs is equally difficult: Section 3.3 explicitly accounts for the additional work of expressing the answer in the required representation.

To address the concern about understanding Redstone rules, C2 reports that Sol succeeds on 10/10 Redstone comprehension probes. These include:

  • Routing: Turning an existing horizontal wire upward to reach a lamp.
  • Branching: Connecting an already-placed second lamp to an existing wire while preserving the first connection.

Both require maintaining wire continuity and the existing layout. These successes support component and spatial comprehension, yet 89% of Sol's complete circuits fail structural checks.

For chess, Sol correctly encodes 55/55 supplied UCI moves into Syn-Chess in one request, given the moving side and physical piece (common response C2), demonstrating that it can apply the encoding rules when the native answer is already available.

The additional work of expressing the answer is part of the representation difficulty we intend to study, not a change in the underlying semantic task.

Q1: Human Baseline

Appendix E provides a complementary human check: two participants achieve 100% and 98% first-move accuracy on 50 puzzles each, given a tutorial similar to the information given in the LLM prompts, showing that the supplied rules can be applied while solving.

ParticipantFirst-move accuracyFull-puzzle accuracy
1100% (50/50)94% (47/50)
298% (49/50)92% (46/50)

Using experienced players and low-rated puzzles reduces the influence of native chess difficulty without first revealing the solutions. We will clarify participant-background and protocol details; this is a small usability study, not a population-level baseline.

Q2: Word Limit

See common response C3. The cap was introduced to avoid long explanations ending without an answer (Appendix C); the native prompt carries the same constraint. On 500 puzzles, removing the limit leaves PRR below 1% for both GPT models. Although placing reasoning before the answer improves PRR, Sol still loses 64.39% and Luna 93.14% of their native successes. We will include these controls in the final version.

W3: Scope & SFT

Our evaluation covers Boolean function implementation and chess notation. Boolean function implementation is not itself game-playing; Redstone is the alternate implementation medium. We agree that two domains do not support a universal transfer and will narrow down our claims.

SFT. Representation-specific training may help, but we question whether a model's existing success transfers without retraining. Vashishtha et al., Executable Counterfactuals (ICLR 2026, Section 4.2), likewise find that SFT's in-distribution gains need not generalize to unseen reasoning structures. We have not tested SFT on PRR.

If these clarifications and additional results address the concerns to the reviewer's satisfaction, we respectfully ask that they consider raising their evaluation score.

Thanks authors for the detailed response. However, my main concerns on the main claim and task equivalence remain. If “constructing the two outputs is equally difficult” cannot be held, why would we expect the two to have similar performance?

I do find the tasks interesting. My suggestion is to revise the claims and the whole story.

Best,

Reviewer_8jdXOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces Paired Representation Robustness (PRR), a metric/methodology for evaluating how sensitive LLMs are to changes in task representation when solving puzzle-like problems. PRR is evaluated as the fraction of "native" successes (examples that the evaluated policy got correct under the original representation) which are retained under an alternative representation. The paper evaluates on Boolean logic as well as chess puzzles, with the chess puzzles providing longer horizon tasks through "Mate-in-k" puzzles. The paper evaluates models in base settings as well as with scaled up inference time compute, finding that task representation significantly impacts success as alternative representations consistently result in low PRR. The paper also finds that introducing routing for scaling inference compute reliably improves PRR across multiple models. The paper additionally provides failure analysis for alternative representations on both evaluated domains.

Critical strengths

The paper's experiments provide clear evidence to the primary claims and do demonstrate high levels of sensitivity in performance with respect to the task representation given to an LLM. The paper provides meaningful ablations like task depth for the Chess domain as well as comprehensive failure analysis.

Critical weaknesses

The paper's experiments may require more alternative representations or domains to back the central claims further. Currently it is logical that PRR would be poor given that the alternative representations consistently have very low Stage 1 success rates. A comparison between more alternative representations per domain would help to disentangle this, especially if some alternative formats provide the some of structural components that the native representation has. On a more minor note, more evaluated models in the primary experiment tables and inference time scaling experiment would provide a stronger claim.

Questions for authors

Have any attempts at evaluation been done where the models are provided in-context examples of utilizing the alternative representation and if so does this context result in improved PRR?

2 comments
AuthorsOpenReview ↗

Response to Reviewer 8jdX

We thank the reviewer for their thorough review and for recognizing the evidence supporting the main claims and the usefulness of the failure analysis.

W1: Step 1 and Representation Transfer

We respectfully disagree that low Step 1 success makes low PRR an expected consequence. Claude retains 66.71% of native successes despite scoring 0% without the rules. Low Step 1 accuracy does not imply low Step 3 retention. As described in Section 3.2, the three steps answer different questions:

  • Step 1: Evaluate the alternate representation without its rules.
  • Step 2: Establish native competence by identifying the exact instances the model already solves.
  • Step 3: Supply the alternate rules and measure how many of those same instances remain correct.

The results directly demonstrate this distinction:

ModelSyn-Chess without rules (Step 1)PRR with rules (Step 3)
GPT-5.6-Sol0%0.69% (6/866)
Claude Opus 4.80%66.71% (493/739)

Appendix E provides a complementary human check: two participants achieve 100% and 98% first-move accuracy on 50 puzzles each, given a tutorial similar to the information given in the LLM prompts, showing that the supplied rules can be applied while solving.

Q: In-Context Examples

We add three solved examples to both native and alternate chess prompts and observe that few-shot prompting does not help recover Syn-Chess retention, while the Native accuracy improves slightly, in this experiment.

ModelPromptNative (Step 2)PRR (Step 3)
GPT-5.6-LunaOriginal56.00% (56/100)1.79% (1/56)
GPT-5.6-LunaThree-shot59.00% (59/100)1.69% (1/59)

We select 100 puzzles by randomly sampling one from each consecutive block of ten, from our full benchmark, to preserve the difficulty coverage. We will provide few-shot ablation across others models too, in the final draft.

W2: Additional Output Representations

We evaluate cipher-only and displacement-only alternative chess representation, and observe that both representation do better than the original, and displacement only notation retains lesser PRR than the cipher-only PRR.

ModelRepresentationNative (Step 2)PRR (Step 3)
GPT-5.6-LunaCipher-only65.00% (65/100)40% (26/65)
GPT-5.6-LunaDisplacement-only65.00% (65/100)4.62% (3/65)
GPT-5.6-LunaOriginal56.00% (56/100)1.79% (1/56)

Such extensions would broaden the evaluation, while PRR's central question remains the same: does demonstrated problem-solving success transfer when the required output representation changes?

W3: Model Coverage

The primary evaluation covers five models and RQ2 covers three; we will prioritize broader closed & open-weights model coverage in the final version.

If these clarifications and additional results address the concerns to the reviewer's satisfaction, we respectfully ask that they consider raising their evaluation score.

Thank you to the authors for the response and my apologies for a delayed reply. I appreciate the clarification on my first weakness and I believe that the final version of this paper would benefit from more analysis into why Opus 4.8 specifically does not follow the trend all other models do despite presumably no advantage in capabilities over a model like GPT 5.6-Sol. With the results from the ablation on alternative representation types I am a bit concerned that the scope of the claim is still not yet enough to meet acceptance, so I retain my score.

Reviewer_7TwYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces Paired Representation Robustness (PRR), the fraction of a model's native-correct problems that remain correct under an alternative output representation. Experiments compare Python with Minecraft Redstone circuits and UCI with a synthetic chess notation. The paper reports large losses of retained success and substantial recovery through additional reasoning on a 100-puzzle Syn-Chess subset, at increased inference cost. Output-interface robustness is a worthwhile problem, but the prompting protocol and missing decomposition and routing controls limit the interpretation of these findings.

Critical strengths
  1. Conditioning on previously correct instances makes lost successes visible even when improvements on other instances would obscure them in aggregate accuracy.
  2. Both domains use semantic correctness targets, and the explicit Syn-Chess transformations offer a useful starting point for controlled interface-robustness tests.
  3. The paper reports substantial model differences, concrete structural failure categories, and inference costs. The small human study also provides a limited usability check for the supplied notation rules.
Critical weaknesses

1.The purported CoT baseline requires the answer before the explanation. The chess prompts in Appendix C explicitly place moves first and reasoning second, limiting the latter to two sentences and fewer than 40 words. Additional reasoning is disabled in the base setting. Visible reasoning therefore cannot assist the first-move prediction before it is emitted. The claim of collapse despite explicit CoT is consequently not adequately supported, and recovery with reasoning enabled may partly reflect removal of this restriction. Please include reasoning-before-answer, sufficient output budgets, solve-then-encode prompting, and budget-matched native or alternate controls. 2.Representation changes are confounded with additional computational tasks. Python-to-Redstone transfer introduces circuit synthesis, spatial routing, support constraints, and Minecraft execution semantics. Syn-Chess is closer to a pure encoding change, but the paper lacks an encoding-only test given the correct UCI answer. Separate solving, encoding, and joint execution, and provide verified reference constructions and staged synthesis/layout baselines for Redstone. The circuit and streaming examples establish that extra resource requirements are possible, but do not identify the cause of the measured LLM failures. 3.The adaptive router is not compared with the necessary fixed-budget baselines. Table 3 includes none, router, and an outcome-informed oracle, but omits always-low/medium/high policies, random routing, and cost-matched repeated sampling. The results show that additional computation helps, not that this router is advantageous or that a greater-than-tenfold cost increase is necessary for transfer. Please report accuracy–cost curves, routing overhead, uncertainty, and evaluations beyond the small chess subset.

Questions for authors

See in Weakness

1 comment
AuthorsOpenReview ↗

Response to Reviewer 7TwY

We thank the reviewer for their thorough review and for recognizing the value of native-success conditioning and the failure analysis.

We ran all three requested controls. With reasoning placed before the answer, Sol still loses 64% and Luna 93% of their native successes (common response C3). Solving, encoding, and joint execution are now separated (common response C2). The fixed-effort baselines show that the decisive step is none → low reasoning, at 5.5–6.2× the cost, while the same puzzles need no reasoning in UCI.

W1: Reasoning Order and Output Budget

See common response C3. On 500 chess puzzles, removing the word limit leaves PRR unchanged (Sol 0.47%, Luna 0.98%). Placing reasoning first raises it to 35.61% and 6.86%, so most native successes are still lost. The native prompt has the same answer-first, 40-word constraint, and Sol solves 86.6% under it. We will relabel the base setting "answer-first" rather than CoT, and add the reasoning-first column to Table 2.

We further tested:

  • Solve-then-encode prompting: We instructed Luna to solve in ordinary chess, then encode and check its moves. At 4K/none, PRR is 3/65 (4.62%), versus 4/65 (6.15%) for the original prompt on the same 65 native-success instances within the seeded 100 puzzles.
  • Output-budget control: Increasing Luna's allowance from 8K to 16K yields PRR of 0/58 (0%) and 1/56 (1.79%), respectively, on each run's native-success subset within the same 100 puzzles. The low PRR persists at 16K even though neither run exhausts its output budget.

W2: Task Preservation and Decomposition

See common response C2, which separates solving, encoding, and joint execution. In chess, Sol with no reasoning solves 86.6% in UCI and encodes 55/55 given moves into Syn-Chess, yet retains only 0.69% when it must do both. This is the coordination-dominated case. In Redstone, the function is given, so the difficulty lies in construction: components are placed correctly in 10/10 probes, but 89% of full circuits fail structural checks. This is the encoding-dominated case.

W3: Fixed-Effort Baselines and Routing Costs

RQ2 asks how much native success additional inference compute can recover, and at what cost. We agree that fixed-effort and repeated-sampling comparisons help establish how that compute should be spent.

Cost-matched repeated sampling. Sampling retains only 1/56 (1.79%), versus the router's 44/56 (78.57%), at approximately the same cost. Using the original 100 Luna prompts and fixed 56 native-success cases, we re-evaluated the first 14 saved none-effort samples per puzzle at temperature 1 with plurality voting. Including native evaluation, costs are $0.156 for sampling versus $0.155 for routing. Even an oracle selecting any correct sample reaches only 3/56 (5.36%). Thus, spending a comparable budget on repeated sampling does not recover the router's retained successes.

Fixed-effort baselines. Always-low reasoning recovers about 98% of native successes for both GPT models. As reported in RQ2, the router increases inference cost from $0.220 to $2.518 for Sol and from $0.010 to $0.154 for Luna. Below, we evaluate always-low, always-medium, and always-high policies.:

PolicySol PRRSol Cost (USD)Luna PRRLuna Cost (USD)
Always-none3.57% (3/84)$0.2201.79% (1/56)$0.010
Always-low98.81% (83/84)$2.65198.21% (55/56)$0.131
Always-medium98.81% (83/84)$3.68889.29% (50/56)$0.225
Always-high100% (84/84)$4.47894.64% (53/56)$0.402
Router79.76% (67/84)$2.51878.57% (44/56)$0.154

These results show that additional inference compute can recover native successes lost at the base setting, while distinguishing recovery from router superiority: always-low retains more native successes than the router for both models. For Luna, always-low also costs less than routing, so the router is not the most efficient policy in this setting. We will include these results in the final version of the paper.

For uncertainty estimates, the router's approximate 95% Wilson intervals are 69.96%-86.96% for Sol and 66.18%-87.29% for Luna.

If these clarifications and additional results address the concerns to the reviewer's satisfaction, we respectfully ask that they consider raising their evaluation score.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Common response

We thank the reviewers for sharing their comments and feedback. Several reviewers raised the same three questions: whether the two representations really pose the same task, whether the answer-first prompt is what causes the collapse, and whether the loss in performance matters if reasoning can recover it. We answer these questions together here and keep the individual replies concise. We begin with a brief example of why output representation matters in practice.

C1. Why output representation matters?

Consider a control engineer who needs a rule that runs a pump whenever either tank sensor trips and the emergency stop is clear. An LLM writes this in Python accurately. The plant controller, however, accepts only ladder logic or a vendor DSL, so the engineer pastes the manual into the prompt. The logic and the truth table are unchanged, yet the model performs far worse in the DSL than in Python. Task difficulty is usually treated as a property of the underlying semantic task. Our results show that the required output form changes both whether the model succeeds and what success costs.

C2. In what sense the tasks are the same, and where the difficulty comes from?

By "same task" we mean the same input, the same set of accepted answers, and the same verifier. We do not claim the two versions are equally hard. Section 3.3 decomposes the cost into three terms: solving the problem (s), expressing an answer once it is available (e), and coordinating the two (h). Below we share our ablations on GPT-5.6-Sol, with none reasoning, to isolate these terms:

Solve aloneEncode aloneBoth together
Chess → Syn-Chess86.6% (866/1000)100% (55/55, correct UCI move given)0.69% (6/866)
Boolean → Redstone98.8% (494/500)100% (10/10 single-component probes)1.01% (5/494)

In chess, each component succeeds on its own, and only the combination fails. This mirrors our streaming example in Sec 3.3. The rules themselves are not the obstacle: two >1000 rated players given the same rules reach 98-100% first-move accuracy.

Redstone shows the other regime. Because the target function is stated in the prompt, little remains to be solved, and the failure lies in construction. GPT-5.6-Sol places individual components correctly in 10 of 10 probes, yet 89% of Sol's complete circuits fail structural checks before their function is even tested.

C3. Does the answer-first prompt explain the collapse?

The base prompt asks for the moves first and caps the explanation at 40 words. We introduced this cap because uncapped runs produced long traces that often ended without an answer (Appendix C). The native prompt carries the same constraint. Under it, Sol solves 86.6% of puzzles, and Claude Opus 4.8 retains 66.71% in Syn-Chess. Because the constraint applies equally to both sides of the pair, it cannot by itself account for the gap. We also tested it directly on 500 chess puzzles, with reasoning set to none and a 4K-token output allowance:

Model40-word limit, moves firstNo limit, moves firstNo limit, reasoning first
GPT-5.6-Luna1.59% (5/314)0.98% (3/306)6.86% (14/204)
GPT-5.6-Sol0.46% (2/435)0.47% (2/427)35.61% (125/351)

Removing the word limit does not help. Letting the model reason before answering does help a bit, but Sol still loses 64% of its native successes, and Luna loses 93%.

C4. What recovery in paired representation robustness costs?

In the chess benchmark, a single level of reasoning restores about 98% of native successes for both GPT models (83/84 and 55/56). However, it costs 5.5–6× as much, while the same puzzles need no reasoning at all in UCI. Cheaper remedies do not work: three-shot prompting reaches 1.69%, cost-matched repeated sampling 1.79%, and a larger output budget 1.79%. The loss is therefore recoverable, but only at a substantial price, and that price is the cost of output representation we set out to measure. The full table appears in our reply to Reviewer 7TwY.

AuthorsOpenReview ↗

Closing Response to the Area Chair

We thank the Area Chair and reviewers. The reviews recognize native-success conditioning and semantic correctness checks (7TwY), clear evidence supporting the main claims (8jdX), the valid chess setting and useful failure analysis (Mzbq), and the interesting case studies and their validation (mVja). Below we summarize our additional evidence and clarifications.

Reviewer concernHow we addressed it
Prompting and word limit (7TwY, mVja)500-puzzle controls: removing the limit alone leaves first-move PRR below 1%. Reasoning-first improves Sol to 35.61% and Luna to 6.86%; we will revise the original CoT characterization.
Task preservation and rule comprehension (7TwY, Mzbq, mVja)Clarified same semantic target, not identical computation; Redstone is implementation transfer. On a separate test, Sol encodes 55/55 supplied moves at none, with side and physical piece given. Redstone probes support basic component/spatial comprehension, not full synthesis competence.
More controlled representations (8jdX)On the same 65 native-correct Luna cases, first-move retention is 26/65 for ciphered-tag/UCI output, 3/65 for displacement-only output, and 4/65 for full Syn-Chess.
Few-shot recovery (Mzbq, 8jdX)Three held-out demonstrations yield Luna first-move PRR of 1/59, versus 1/56 without them, using each condition's native-success set. They do not recover retention in this experiment.
Recovery baselines and costs (7TwY, Mzbq)Low reasoning retains 83/84 Sol and 55/56 Luna native successes. On Luna's fixed gate, approximately cost-matched sampling retains 1/56, versus routing's 44/56. Fixed-low is also cheaper and more accurate than Luna's router; we do not claim optimal routing or a necessary tenfold cost increase.
Human baseline and scope (mVja, Mzbq, 8jdX)Two chess players achieve 50/50 and 49/50 correct first moves after instruction: a usability check, not a population baseline. We will limit claims to the evaluated models/settings and remove the shortcut-learning attribution.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.