- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies how curriculum, model capacity, and training-trace format affect tool-based graph navigation. A change to ordering-direction coverage reverses the relative performance of compiled navigation and free-form reasoning. Experiments also separate tool participation from task success and test transfer to a relabeled organizational domain. The synthetic graph allows the training curriculum to be changed while keeping the underlying navigation task fixed.
Critical strengths
-
The curriculum intervention produces a effective result: navigation leads free-form reasoning by 23.7 pp on Gemma-12B and 13.9 pp on Qwen3-14B after the change. This is useful evidence that training coverage can determine the apparent benefit of a reasoning format.
-
The paper reports negative transfer results and abstention trade-offs rather than relying only on a favorable aggregate metric. These analyses make the central curriculum finding more informative. Separating tool participation from grounded answer correctness helps reveal why aggregate success changes.
Critical weaknesses
-
The format effect is not examined cleanly. Under token-based matching, the tool-matched prose control B′ covers only 49% of the training examples in D. So the evidence for compiled notation itself is weaker than that for curriculum coverage. Comparing both example-matched and token-matched variants would help quantify this confound instead of attributing the entire difference to representation.
-
Transfer is tested through a narrow, largely structure-preserving domain relabeling. Together with the failed transfer advantage, this limits conclusions about learning reusable navigation procedures beyond the synthetic task. Besides, whether the learned procedure would survive changes in graph structure, relation composition, or the number of required tool steps.
Questions for authors
Does the D–B′ difference persist when unique training-item coverage is matched, with token budget reported separately?
Thank you for the thoughtful rebuttal, and for running the follow-up B' experiment on Gemma-12B.
W1: Adding the end-to-end overview figure significantly improves readability. W2/W3/Q1: I appreciate the explicit clarifications regarding the diagnostic role of B' and the scope boundaries around closed-form synthetic graph queries versus open-ended, real-world agent environments. Q2: Thank you for testing your train/evaluation mismatch hypothesis directly and transparently reporting both the partial recovery and the remaining unexplained residual.
All of my concerns have either been constructively addressed or properly acknowledged. I will update my Presentation score and maintain overall recommendation in support.