- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper compares protocol stacks, supervised fine-tuning, and structural-reward GRPO on a 100-task financial tool-use suite across several Qwen model sizes. It reports large protocol effects, heterogeneous SFT effects, small RL point differences, and case studies involving a mismatched tokenizer template and SFT-induced tool-call formatting failures.
Critical strengths
The tokenizer-template investigation and the 4B behavioral analysis identify practically important evaluation and training failures. In particular, separating format repair from repeated use of an always-successful tool explains why a policy can change substantially without improving the measured final-answer rate.
Critical weaknesses
W1. The pooled McNemar analysis uses dependent pairs. Section 3 combines all nine cross-seed pairings per task. Each output is reused three times, so the 900 pairs are not independent observations. The stated exact p-values are not valid for this construction. Non-significance also does not establish that the RL effect is negligible.
W2. The metrics do not establish task-level correctness. A well-formed final answer can be wrong, and numbers traceable to source material can still be irrelevant to the requested computation. These metrics measure formatting and numerical provenance, not successful task completion. They do not support the broader attribution of task capability to protocol, SFT, and RL.
Questions for authors
-
What are the effect sizes and confidence intervals when the analysis retains all seeds from each task together, rather than treating cross-seed pairings as independent observations?
-
What magnitude of RL improvement can the data exclude? Please specify a practically meaningful equivalence margin and assess whether the uncertainty interval lies within it.
-
What is the template-by-training interaction: the difference between the pre/post-training changes under the correct and incorrect templates? A significant change under one template and a non-significant change under another does not establish that the changes differ.
-
On a blinded sample of existing outputs graded for correctness on the actual task, do the protocol, SFT, and RL comparisons retain the reported ordering?
-
The RL reward emphasizes structural success. What evidence distinguishes a limitation of this reward and training setup from a limitation of RL for tool-use learning? Which conclusions are intended to apply only to the tested setup?