- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
The paper proposed an interesting analysis between so called ``same-task'' (bool -> Minecraft, UCI -> syn-chess) has difficult representation transfer. It claimed that the transfer fails from the two experiments and need more than 10 times cost reasoning to recover the gain.
Critical strengths
- The topic is interesting and the two case study are interesting
- The experiments are validated via baseline comparison and human annotation
Critical weaknesses
I have significant concerns about the basis task definition of the paper so before this has been clarified, I am hard to accept the following experiments and claims:
Most importantly, the paper claims that you have same task but essentially the tasks are not the same, and even these are quite different. Boolean task is just a combination task but while you want to create items in minecraft, you need to know basic knowledge in minecraft. It is true that you may give some rules but there are a lot of reasoning behind for example you need to understand the materials, goods, and even what means the combination in minecraft! The same is true for the the second case: - syn-chess needs understand the rules, calculating the moving distance, which in the prompt you banned this skills. I could only admit that there are indeed some skills are shared but definitely not the same tasks. Therefore, the two case you found would not support the claim ``LLM representation does not transfer'' as they might transfer, and the failure is due to the task they perform needing more skills.
So you may need to test a more controllable application pair such as - you need to compare boolean representation with a de-gamified grid-circuit representation - something similar.
Despite all above,
- You force reasoning less than 40 words, and the RQ2 allows that, which gives a results from 3% to 79%. It is a sign that the task needs explicit reasoning but not difficulty of transfer
- You are trying to argue against the general transfer difficulty but only two fields are given. I understand exhaustively running through all cases is impossible but I would suggest select one field - for example game? health? math? and narrow down your claim
- Only 100 tests are given for RQ2
- It could be better to explain Claude Opus 4.8 has 66.71% on Syn-Chess but GPT is less than 2%.
- You need to give demographics for human participants and their knowledge on chess and minecraft.
Given the above concerns, I do not think the paper could be revised during this round. I therefore recommend rejection.
Questions for authors
See above please.
4 comments
Response
Thanks the author response. It is nice to hear the authors are willing to do additional experiments for human annotation and my concerns for W3 and W4.
For W1, it is still not convincing - that the two tasks are same - so if you mean the sub-tasks are the same, the overall contribution should be narrowed for description and then in which sense the contribution will be limited. And i am confused by We do not claim that the two representations are equally difficult. .. the meaning of difficult is no clear.
In conclusion - all the concerns are not resolved so due to the limited time, I maintain my score and suggest the paper to be resubmitted :).
Follow-up Response to Reviewer mVja
We thank the reviewer for their quick follow-up and the opportunity to clarify W1. Below we clarify the definitions and the scope of our work as per your request.
Definition Clarifications
Below we share an example to explain what we mean by "same task"
In chess domain:
Consider one fixed chess position where an accepted first move takes White's rook from c1 to c8. UCI writes c1c8; Syn-Chess writes Qc1 B 7. Both specify the same move on the same board. The model sees the same board in both conditions and must solve it each time. Both answers (the model’s first move) are converted back into a board move and checked against the same list of correct moves. Thus, only the way the answer is written differs.
By "difficulty," we mean the additional work of expressing a solution: here, calculating displacement and applying the cipher. We do not mean a different mating objective or identical computation in the two conditions. This is consistent with the distinction between informational and computational equivalence in Larkin and Simon (1987), Why a Diagram Is (Sometimes) Worth Ten Thousand Words: equivalent information need not support equally efficient processing.
Finally, our contribution is not to establish that these extra operations exist, but to measure how much demonstrated success survives each output requirement. The new controls make this distinction concrete. On the same 65 native-first-move-correct Luna instances within the 100-puzzle sample, with none reasoning and a 4K allowance:
| Output requirement | Retained native first-move successes |
|---|---|
| Ciphered piece tag, unchanged UCI coordinates | 26/65 (40.00%) |
| Displacement-only notation | 3/65 (4.62%) |
| Full Syn-Chess | 4/65 (6.15%) |
Retention remains low in both conditions requiring displacement output. These results quantify which output requirements retain native success; they do not establish that the model's underlying chess skill disappeared.
In Redstone domain:
the specified Boolean function and its truth table remain fixed, but producing the output additionally requires component knowledge, circuit synthesis, and spatial layout.
Scope Clarifications
- We will distinguish Redstone's less-controlled implementation-transfer setting from the chess notation change, rather than present both as pure symbol recoding.
- We will replace unqualified "same task" with "same semantic task" in the title and claims. The contribution will be stated as retained correctness under changed output requirements on instances the model already solves, not equivalence of complete computational tasks or a general inability of LLMs to transfer.
We hope this example and the clarified scope address the reviewer's remaining concern.
Response
I thank the authors efforts.
W1. This is still the same sub-tasks if I may ... and the example does not fully resolved my concerns as in this example, the latter one contains additional movement notation. In another example, the redstone has game knowledge which is overlooked by this response.
The rest W3,W4 have not been resolved. Only when the results could be made, and we see the example, we could get the chance to see whether it has been resolved.
Therefore, I maintain my score. I hope this active discussion has explain my position and I will leave it to AC for their decision.
Response to Reviewer mVja
We thank the reviewer for their thorough review and for recognizing the value of the two case studies and their validation.
See common response C2, which clarifies task preservation and separates solving, encoding, and coordinating the two. Python and Redstone must implement the same Boolean function; UCI and Syn-Chess must solve the same position. We do not claim that the two representations are equally difficult. The additional skills raised by the reviewer are the construction and coordination requirements explicitly discussed in Section 3.3. PRR asks whether demonstrated native success survives these output requirements.
Regarding "calculating the moving distance," the prompt does not ban this skill or the use of ordinary chess knowledge. Standard chess rules and board states remain unchanged; only the final notation changes.
Redstone is already presented through an ARC-AGI-inspired symbolic matrix, with component behavior and placement examples supplied, rather than interactive gameplay. A fully de-gamified grid-circuit comparison would be a useful further control; we do not equate the existing matrix interface with that additional experiment.
See common response C2. Given the UCI move, moving side, and physical piece, Sol correctly encodes 55/55 moves into Syn-Chess in one request. Sol & Luna also succeed on 10/10 & 9/10 Redstone comprehension probes, including turning an existing wire toward a lamp and branching it to a second lamp while preserving the original connection. These test wire continuity and spatial layout, not just component names, supporting comprehension of the minecraft supplied rules.
Appendix E provides a complementary human check: two participants achieve 100% and 98% first-move accuracy on 50 puzzles each, given a tutorial similar to the information given in the LLM prompts, showing that the supplied rules can be applied while solving.
Using experienced players and low-rated puzzles reduces the influence of native chess difficulty without first revealing the solutions. We will clarify participant-background and protocol details; this is a small usability study, not a population-level baseline.
See common response C3. On 500 puzzles, removing the 40-word limit leaves PRR below 1% for both GPT models. Removing the limit and placing reasoning before the answer improves transfer, but Sol still loses 64.39% and Luna 93.14% of their native successes. Thus, the word limit alone does not explain the collapse. We will include these controls in our final draft.
RQ2 retains the same visible 40-word restriction and changes API reasoning effort, so its recovery is not obtained by lifting that restriction. Recovery with additional compute addresses the paper's question, rather than contradicting it: how much native success can be recovered, and at what cost? (common response C4)
We thank the reviewer for this suggestion and will include an in-depth analysis of Claude's successes and failures in the final draft to better understand the Claude-GPT performance gap. We agree that two domains do not support a universal transfer and will narrow down our claims.
For RQ2, we sampled one puzzle from each consecutive block of ten after sorting the 1,000 puzzles by rating, ensuring coverage across the benchmark's difficulty range. We will clarify this sampling procedure in the paper.
If these clarifications and additional results address the concerns to the reviewer's satisfaction, we respectfully ask that they consider raising their evaluation score.