acceptodds
Under review as a conference paper at ICLR 2027

The Protocol Is the Agent: Decomposing RL, SFT, and Scaffold Effects in Tool-Use Agents from 0.8B to 27B

Abstract

On a single frozen instrument (the same 100 financial tool-agent tasks, execution harness, and analyzer, with three evaluation seeds per arm and pooled paired exact McNemar tests), we decompose the measured task-completion capability of one Qwen3.x family (0.8B, 2B, 9B, 27B, plus a 4B case study) into protocol, supervised-fine-tuning (SFT), and reinforcement-learning (RL) contributions. Three findings. (1) A consistent RL null: the same GRPO recipe with verifiable structural rewards changes nothing at any scale (+0.7 at 0.8B, p=.75; +2.0 at 2B, p=.27; 0.0 at 9B, p=1.0; -0.7 at 27B, p=.64), including the scale with the largest SFT headroom. We further document how a shared-but-wrong evaluation template alone manufactured p=.024 in our own 2B cell before correction. (2) The protocol dominates: switching the surrounding stack from a text-protocol scaffold to native tool-calling is worth +59 points on the same 2B SFT weights, versus +4.3 for SFT itself and +2.0 for RL. (3) SFT's contribution is non-monotone and can be catastrophic: the identical recipe adds +18 points at 0.8B and +11 at 9B but destroys 45 points at 4B by inducing temperature-dependent format fragility in an otherwise format-clean base, after which RL repairs the format while capturing a rewarded shortcut. Corollaries: measure the protocol before the weights, the headroom before the RL budget, and never compare training arms without tokenizer-exact evaluation.

Primary area:
foundation or frontier models, including LLMs
Keywords:
reinforcement learning with verifiable rewards, tool-use agents, supervised fine-tuning, agent evaluation, financial agents, capability attribution

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_EdpQOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper compares protocol stacks, supervised fine-tuning, and structural-reward GRPO on a 100-task financial tool-use suite across several Qwen model sizes. It reports large protocol effects, heterogeneous SFT effects, small RL point differences, and case studies involving a mismatched tokenizer template and SFT-induced tool-call formatting failures.

Critical strengths

The tokenizer-template investigation and the 4B behavioral analysis identify practically important evaluation and training failures. In particular, separating format repair from repeated use of an always-successful tool explains why a policy can change substantially without improving the measured final-answer rate.

Critical weaknesses

W1. The pooled McNemar analysis uses dependent pairs. Section 3 combines all nine cross-seed pairings per task. Each output is reused three times, so the 900 pairs are not independent observations. The stated exact p-values are not valid for this construction. Non-significance also does not establish that the RL effect is negligible.

W2. The metrics do not establish task-level correctness. A well-formed final answer can be wrong, and numbers traceable to source material can still be irrelevant to the requested computation. These metrics measure formatting and numerical provenance, not successful task completion. They do not support the broader attribution of task capability to protocol, SFT, and RL.

Questions for authors
  • What are the effect sizes and confidence intervals when the analysis retains all seeds from each task together, rather than treating cross-seed pairings as independent observations?

  • What magnitude of RL improvement can the data exclude? Please specify a practically meaningful equivalence margin and assess whether the uncertainty interval lies within it.

  • What is the template-by-training interaction: the difference between the pre/post-training changes under the correct and incorrect templates? A significant change under one template and a non-significant change under another does not establish that the changes differ.

  • On a blinded sample of existing outputs graded for correctness on the actual task, do the protocol, SFT, and RL comparisons retain the reported ordering?

  • The RL reward emphasizes structural success. What evidence distinguishes a limitation of this reward and training setup from a limitation of RL for tool-use learning? Which conclusions are intended to apply only to the tested setup?

Reviewer_EExDOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper decomposes a tool-use agent's measured performance into contributions from the protocol, SFT, and RL. The primary metric records whether a trajectory ends in a well-formed final answer and does not grade correctness, and a second layer checks that the numbers in the answer are grounded in tool outputs. Qwen3.5 and 3.8 series models at five sizes are compared zero-shot, after LoRA SFT and after GRPO with a structural reward, using three evaluation seeds and an exact McNemar test pooled over seed pairings. The paper reports four findings. GRPO yields no significant change at the headline sizes. Switching from a text-protocol scaffold to native tool calling is worth about fifty-nine points, against a few points for SFT and RL. A wrong chat template produced a spurious RL effect. At the middle size, SFT breaks protocol compliance.

Critical strengths
  1. The paper reports a retracted template result, a single-seed gain that later seeds did not reproduce, two infrastructure faults, and a pre-registered hypothesis that the zero-shot runs falsified. Its practical recommendations are concrete and useful for the community: pin the tokenizer in every evaluation command, use several seeds, log per-row environment errors, and probe for malformed calls at sampling temperature before trusting an SFT checkpoint.

  2. The numeric-grounding layer is a good-faith step beyond a pure format metric, and the paper reports it for every cell rather than only where it helps.

Critical weaknesses
  1. The single decision test overstates significance. Each evaluation run appears in three of the nine pooled seed pairings, so the pairings are not independent and every z-score is inflated by about the square root of three. The reported p-values reproduce exactly from the pooled counts, which confirms that this is how they were computed. With the task as the unit, the template exhibit's significant RL effect falls to a p-value of roughly eighteen percent. The grounded-layer GRPO gain at the middle size and the GRPO gain on the earlier-generation largest model also lose significance. Only the large SFT effects survive. The comparisons need a task-clustered test, such as a paired permutation test over tasks.

  2. The scope is narrow, and the cells that break the pattern are moved out of the headline. The evidence covers one in-house domain, one model family and LoRA only. In the appendix, GRPO gives a nominally significant five-point gain on the earlier-generation largest model, which contradicts the abstract. Two of the four claimed replication cells are the headline cells themselves. Several numbers are also inconsistent with the per-seed tables. For example, the zero-shot tier row of the smallest model averages about forty-one, against a reported overall rate of about forty-five.

Questions for authors
  1. Is the middle-size SFT collapse reproducible across training seeds? One or two additional SFT training seeds at that size would suffice.
  2. What exactly is the GRPO reward, including each term, its weight, whether there is an outcome term, and what the maximum of 11 corresponds to? What fraction of GRPO groups had zero reward variance at each size?
Reviewer_SuDwOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper decomposes financial tool-agent task performance into protocol, supervised fine-tuning, and reinforcement learning contributions using a controlled evaluation of Qwen3.x models at multiple scales. The results show that protocol design has the largest impact, with native tool calling improving performance, while GRPO-based RL yields no statistically significant gains across model scales.

Critical strengths
  1. This paper investigates 100 financial tool-agent tasks and decomposes the task completion capability of one Qwen family into protocol, supervised fine-tuning, and reinforcement learning contributions. This work is interesting and of practical value.
Critical weaknesses
  1. The adopted metric focuses on response formatting and termination rather than actual task-solving capability.
  2. The protocol contribution is not fully identifiable from the described experiments. Protocol effects may couple with SFT and reinforcement learning.
  3. The RL experiment tests a specific recipe (GRPO), not RL in general. More alternative RL methods should be considered.
  4. The 100 financial tool-agent tasks cover several useful financial-analysis scenarios, but may not capture the full diversity of real-world financial workflows.
Questions for authors

Please refer to Critical Weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.