- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper introduces AdviSD, a framework for training a small advisor model to guide a frozen, closed-source LLM executor through natural-language advice. A reflector identifies decisions requiring correction and provides feedback. Selected decisions receive self-distillation supervision from a feedback-conditioned copy of the pre-update advisor. Selection uses the advisor’s log-likelihood contrast for the recorded executor response with versus without advice, while reflection-flagged abstentions bypass this filter. The advisor is optimized jointly with GRPO and self-distillation.
Critical strengths
- The paper presents extensive experiments. The main results support the effectiveness of the proposed method, while comprehensive ablations clarify the contributions of its key components.
- The trained advisor demonstrates strong transferability and generalization, remaining effective across executors (Figure 2) and on out-of-domain tasks (Table 3).
Critical weaknesses
- The executors use default API decoding settings, but their sampling temperatures are not explicitly reported. If nonzero temperatures are used, executor stochasticity may increase reward variance and complicate credit assignment to the advisor during training. Reporting the decoding parameters and comparing against temperature-zero execution, where supported, would clarify the impact on training stability and evaluation reliability.
- The experiments use Qwen3-8B as the sole trainable advisor backbone. Although the paper evaluates transfer across executors, it remains unclear whether the training method yields consistent gains with other open-weight advisor models. Evaluating additional model families and sizes would strengthen the evidence for its general applicability.
Questions for authors
-
What sampling temperatures were used for the executors during training and evaluation? How does setting the executor temperature to zero affect training stability and final performance?
-
Does AdviSD yield consistent improvements when using open-weight models other than Qwen3-8B as the advisor? Could you provide results with additional model families or sizes?
3 comments
Response to W1/Q1
W1/Q1. AdviSD retains a 4.2–4.6-point advantage over GRPO in all four temperature combinations tested with the Claude executor.
Decoding settings. In the main experiments, both executors used the providers' default decoding in training and evaluation. Claude Sonnet 4.6 used temperature 1.0, with top-p and top-k at the serving defaults. Gemini 3.7 Flash used its backend's default decoding, which Google lists as temperature 1.0, top-p 0.95 and top-k 64 [1]. Google also documents that the backend ignores requested values for these parameters [2]. The trainable Qwen3-8B advisor sampled at temperature 0.7, with top-p 1 and top-k disabled, throughout (Table 4).
Executor randomness and credit assignment. Executor sampling affects episode rewards and the trajectories that hindsight feedback reviews. AdviSD computes both contrast terms on the same recorded executor response, so selection adds no executor sampling. Its auxiliary supervision, however, still depends on the realized interaction. We therefore tested whether its benefit persists when the executor temperature changes in training, evaluation or both.
Temperature study. We ran this study with Claude Sonnet 4.6 on BFCL-v3. Because Gemini 3.7 Flash ignores the requested temperature, varying it is not allowed for Gemini 3.7 Flash. The adaptation temperature is the executor's temperature during GEPA's prompt optimization or during advisor training; advisor sampling stays the same. GEPA, GRPO and AdviSD cover all four combinations, while SDPO and DistIL were evaluated only after adaptation at 1.0. Results are mean ± sample SD over three training or prompt-optimization runs, each averaging four test evaluations; fixed baselines use three groups of four evaluations.
| Method | Adapted at 1.0, eval at 1.0 | Adapted at 1.0, eval at 0 | Adapted at 0, eval at 0 | Adapted at 0, eval at 1.0 |
|---|---|---|---|---|
| No advisor | 54.9 ± 0.9 | 53.0 ± 0.8 | 53.0 ± 0.8 | 54.9 ± 0.9 |
| GEPA on executor prompt | 58.0 ± 0.8 | 54.0 ± 0.6 | 55.4 ± 0.7 | 54.4 ± 0.6 |
| Claude Sonnet 4.6 advisor | 60.4 ± 0.7 | 58.1 ± 0.4 | 58.1 ± 0.4 | 60.4 ± 0.7 |
| Untrained Qwen3-8B advisor | 54.2 ± 1.3 | 51.1 ± 0.9 | 51.1 ± 0.9 | 54.2 ± 1.3 |
| GRPO | 58.9 ± 0.7 | 55.2 ± 0.4 | 56.6 ± 0.9 | 55.5 ± 0.4 |
| SDPO | 55.8 ± 0.7 | 51.6 ± 0.5 | – | – |
| DistIL | 57.3 ± 1.3 | 53.5 ± 1.1 | – | – |
| AdviSD (ours) | 63.1 ± 0.6 | 59.8 ± 0.8 | 61.2 ± 0.9 | 60.1 ± 0.8 |
Fixed baselines have no adaptation temperature, so columns with the same evaluation temperature repeat one estimate rather than reporting additional runs. Their SDs summarize variation across three groups of four evaluations, including at temperature 0, where API outputs are still not guaranteed to be identical. A dash means not evaluated.
Findings.
- AdviSD's advantage persists across temperature settings. Its margins over GRPO are 4.2, 4.6, 4.6 and 4.6 points, and it beats GRPO in every BFCL-v3 category in all four settings. With temperature 0 in both adaptation and evaluation, AdviSD reaches 61.2 ± 0.9, against 56.6 ± 0.9 for GRPO and 53.0 ± 0.8 without an advisor. At executor temperature 0, credit still has to be assigned across turns, and fitting a correction can change the advisor without improving execution (Lemma 1). AdviSD's advantage persists in this setting, consistent with the motivation for selecting supervision by its relevance to execution.
- Matching adaptation and evaluation temperatures helps in these experiments. At each evaluation temperature, GEPA, GRPO and AdviSD score higher when adapted at the matching temperature. AdviSD improves over the executor alone by 8.2 points in both matched settings, against 6.8 and 5.2 under a mismatch. It also beats the strongest baseline, the Claude Sonnet 4.6 advisor, by 2.7 and 3.1 points in the matched settings and trails it by 0.3 only when adapted at 0 and evaluated at 1.0. This is consistent with adaptation to the executor's response distribution, which changes with temperature.
- Final-score variability. At evaluation temperature 0, the observed across-run SD is 0.9 for both GRPO and AdviSD adapted at 0, compared with 0.4 and 0.8 when adapted at 1.0. These three-run estimates show no reduction in final-score variability in this comparison. These SDs describe how final scores vary across runs; they do not measure within-run training stability.
We have added this study to the revised paper: Section 7 (Evaluation) summarizes it, and Appendix E.3 (Table 8) gives the decoding settings, protocol and full per-category results.
[1] https://docs.cloud.google.com/gemini-enterprise-agent-platform/models/gemini/3-7-flash
[2] https://docs.cloud.google.com/gemini-enterprise-agent-platform/models/guides/gemini-3-7-flash
Response to W2/Q2
W2/Q2. AdviSD's gains persist with smaller Qwen3-4B and Llama-3.2-3B-Instruct advisors across both executors.
We trained two smaller open-weight advisors: Qwen3-4B, from the same family as Qwen3-8B, and Llama-3.2-3B-Instruct, from a different one. Both follow the training and evaluation protocol of Tables 1 and 3, and every method keeps its Qwen3-8B hyperparameters. Trained methods use three training runs, with four test evaluations per selected checkpoint. AdviSD calibrates its gate threshold separately in each run, using a training-task pilot, and keeps it fixed during training (Eq. 4). OOD evaluation reuses the BFCL-trained checkpoints without further training.
The table gives an aggregate view: the BFCL-v3 average and EnvScaler score (mean ± sample SD, as in Table 1) and the OOD macro average (as in Table 3). Appendix E.7 (Tables 12 and 13) reports every BFCL-v3 category and OOD benchmark component for the new advisors, as Tables 1 and 3 do for Qwen3-8B. Gemini and Claude denote the Gemini 3.7 Flash and Claude Sonnet 4.6 executors. The first three rows are shared controls that do not depend on the trainable advisor backbone. The API advisor is the executor's own model, used as a frozen advisor.
| Advisor | Method | BFCL-v3, Gemini | BFCL-v3, Claude | EnvScaler, Gemini | EnvScaler, Claude | OOD, Gemini | OOD, Claude |
|---|---|---|---|---|---|---|---|
| Shared | No advisor | 57.8 ± 1.4 | 54.9 ± 0.9 | 75.7 ± 2.0 | 77.4 ± 2.1 | 74.5 | 70.4 |
| GEPA on executor prompt | 59.4 ± 0.3 | 58.0 ± 0.8 | 76.0 ± 1.8 | 78.4 ± 1.3 | 72.3 | 68.2 | |
| API advisor | 59.2 ± 0.5 | 60.4 ± 0.7 | 77.7 ± 0.7 | 78.7 ± 0.5 | 74.1 | 70.8 | |
| Qwen3-8B (paper) | Untrained | 56.1 ± 1.0 | 54.2 ± 1.3 | 75.6 ± 2.2 | 74.1 ± 1.5 | 72.1 | 67.2 |
| GRPO | 61.4 ± 1.0 | 58.9 ± 0.7 | 79.7 ± 2.2 | 80.7 ± 2.3 | 75.2 | 71.3 | |
| SDPO | 57.8 ± 0.8 | 55.8 ± 0.7 | 76.0 ± 1.7 | 78.9 ± 1.2 | 72.3 | 68.9 | |
| DistIL | 58.5 ± 0.4 | 57.3 ± 1.3 | 76.1 ± 2.4 | 78.9 ± 1.5 | 73.3 | 68.4 | |
| AdviSD (ours) | 67.8 ± 1.6 | 63.1 ± 0.6 | 84.8 ± 1.6 | 84.6 ± 1.1 | 77.2 | 74.0 | |
| Qwen3-4B | Untrained | 54.6 ± 1.2 | 52.9 ± 1.2 | 74.1 ± 1.7 | 72.6 ± 1.8 | 71.0 | 65.7 |
| GRPO | 60.1 ± 1.0 | 57.1 ± 1.4 | 77.9 ± 2.1 | 78.9 ± 1.6 | 74.8 | 70.7 | |
| SDPO | 56.1 ± 1.1 | 54.7 ± 0.9 | 74.8 ± 1.2 | 77.3 ± 1.0 | 70.8 | 67.1 | |
| DistIL | 57.7 ± 0.7 | 56.2 ± 1.1 | 74.9 ± 1.6 | 77.5 ± 1.2 | 72.2 | 67.3 | |
| AdviSD (ours) | 66.0 ± 1.9 | 62.1 ± 0.9 | 83.4 ± 1.3 | 83.4 ± 1.5 | 76.2 | 72.8 | |
| Llama-3.2-3B-Instruct | Untrained | 53.4 ± 1.0 | 52.6 ± 0.6 | 72.4 ± 0.5 | 71.7 ± 1.3 | 70.3 | 65.0 |
| GRPO | 58.8 ± 0.8 | 57.8 ± 0.9 | 76.2 ± 1.1 | 78.6 ± 0.3 | 73.1 | 70.1 | |
| SDPO | 55.3 ± 0.8 | 53.5 ± 0.8 | 72.5 ± 0.8 | 76.2 ± 1.1 | 70.2 | 66.6 | |
| DistIL | 56.4 ± 0.3 | 54.6 ± 0.2 | 74.7 ± 0.7 | 76.4 ± 0.8 | 71.3 | 66.4 | |
| AdviSD (ours) | 65.4 ± 0.6 | 61.0 ± 1.3 | 80.8 ± 0.9 | 83.8 ± 0.6 | 75.6 | 72.1 |
- AdviSD leads with every backbone. For each backbone and executor, AdviSD has the highest mean on all three aggregate metrics, compared with the shared controls and every method using the same backbone. With the new advisors, it beats GRPO by 3.2–6.6 percentage points on BFCL-v3, 4.5–5.5 score points on EnvScaler and 1.4–2.5 points on the OOD average. The corresponding Qwen3-8B margins are 4.2–6.4, 3.9–5.1 and 2.0–2.7. With both executors, the smaller AdviSD advisors also have higher means on all three aggregate metrics than every Qwen3-8B baseline, including GRPO.
- Training makes the smaller advisors useful. Without task-specific training, both new advisors score 1.6–5.7 points below the executor alone on every aggregate metric. After AdviSD training, both exceed it on all three aggregates with both executors, as with Qwen3-8B.
- The baseline ordering persists. SDPO and DistIL trail GRPO on every aggregate metric for both new backbones, as they do for Qwen3-8B.
Together with the Qwen3-8B results, these experiments show consistent aggregate gains across two advisor families, three model sizes and both frozen executors.
We have added these results to the revised paper: Section 7 (Q1) summarizes them, and Appendix E.7 gives the full tables (Tables 12 and 13).
Thank you again for your thoughtful review, questions and positive assessment of our work. In response to your feedback, we an experiments at different executor temperatures and trained Qwen3-4B and Llama-3.2-3B-Instruct advisors, both of which strengthened the paper. We would be happy to answer any further questions. If our responses and new results resolve your concerns, we would be very grateful if you would reconsider your assessment.
Thank you for your positive assessment and for highlighting the paper's "extensive experiments" and "comprehensive ablations", and the trained advisor's "strong transferability and generalization" across executors and on out-of-domain tasks. Your feedback has helped us improve the paper, and we address the points you raised below. In brief:
The corresponding additions to the revised paper are marked in blue.