- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This work studies online selection among preconstructed prompt and parameter configurations when earlier choices may affect subsequent deployment states. It compares text-only, parameter-only, and joint adaptation by decomposing excess loss relative to the best fixed joint configuration into within-family selection regret, family restriction, and signed state effects. This work combines exponential weights with low-switch sampling to derive conditional regret guarantees and examines the additional assumptions and costs of weaker feedback. Controlled experiments and Qwen3.5-9B evaluations on Spider and SParC illustrate how candidate availability and deployment history affect performance.
Critical strengths
-
This work separately accounts for selection regret, candidate-family restriction, and state effects, providing a useful basis for comparing different adaptation modes.
-
By matching the marginal distributions of independent and lazy sampling, this work isolates deployment-history effects while holding expected diagonal selection loss fixed. This explains why reducing switches can either improve or worsen performance.
Critical weaknesses
-
The Spider experiments and restricted-feedback SQL replays use one fixed evaluation order per benchmark. Because online selectors update from preceding observations, cumulative loss and the relative ranking of selectors may depend on this order. Replaying the existing Spider and cleared-history SParC loss matrices under several task orders would help establish the robustness of the reported comparisons.
-
The sensitivity of the restricted-feedback results to the online selection learning rate remains unclear. This work compares joint and coordinatewise selection under different learning rates, but the feedback comparisons in Appendices C.5 and C.9 primarily use the theory-prescribed rate for each regime, without reporting a systematic sensitivity analysis around these settings.
Questions for authors
Please address the concerns raised in Weaknesses
4 comments
The sensitivity of the restricted-feedback results to the online selection learning rate remains unclear. This work compares joint and coordinatewise selection under different learning rates, but the feedback comparisons in Appendices C.5 and C.9 primarily use the theory-prescribed rate for each regime, without reporting a systematic sensitivity analysis around these settings.
We added a systematic learning-rate sweep for full- and restricted-feedback methods and comparisons at shared numerical rates. We also report development-tuned full EW. Section 7 summarizes the findings, and Appendix F.3 reports the complete grids, tuning protocol, and uncertainty estimates. The theoretical bounds retain the learning rate explicitly; the sweeps examine sensitivity around the rates that balance those bounds.
We swept multipliers , with a separate zero-rate diagnostic, for the full- and restricted-feedback methods. The SQL analysis uses the three models, nine seeds, and predefined subsets described above. The table gives mean joint-family held-out failures across nine seeds; each arrow compares a method's prescribed rate with eight times that rate. Sparse EW draws each deployment choice afresh from its current probabilities, and deployment-only selection uses block length 1. For the last column, each bank's EW multiplier minimizes average development loss over the original order and four development permutations. We freeze the multiplier and rate formula, then evaluate the numerical rate at the held-out horizon.
| Model | Benchmark | Full EW: prescribed → ×8 | Sparse EW: prescribed → ×8 | Deployment-only: prescribed → ×8 | Development-tuned full EW |
|---|---|---|---|---|---|
| Qwen3.5-9B | Spider | 15.504 → 15.755 | 14.894 → 14.899 | 14.880 → 14.865 | 15.173 |
| Qwen3.5-4B | Spider | 17.447 → 16.936 | 16.724 → 16.729 | 16.732 → 16.781 | 16.761 |
| Ministral 3 8B | Spider | 14.899 → 14.419 | 16.719 → 16.658 | 16.605 → 16.507 | 14.539 |
| Qwen3.5-9B | Cleared SParC | 51.880 → 52.489 | 51.920 → 51.872 | 51.963 → 51.706 | 52.185 |
| Qwen3.5-4B | Cleared SParC | 54.721 → 53.783 | 57.707 → 57.403 | 57.831 → 56.911 | 54.079 |
| Ministral 3 8B | Cleared SParC | 58.614 → 58.253 | 63.779 → 63.087 | 63.793 → 62.530 | 58.234 |
Development tuning lowers mean held-out loss in five of six settings; Qwen3.5-9B cleared SParC is the exception. This evaluation uses only development outcomes to choose the multiplier, while the held-out sweep describes sensitivity.
Shared numerical rates separate the effect of feedback from the rate prescription. On Qwen3.5-9B Spider, full EW, sparse EW, and deployment-only selection yield 14.935, 14.894, and 14.880 failures at the sparse-diagonal rate, versus 15.504, 14.899, and 14.865 at the full-diagonal rate. Rate choice therefore explains part of the observed separation. At the sparse-rate anchor, the conditional Monte Carlo standard errors of the sparse and deployment-only means are 0.005 and 0.037 failures, respectively; the full EW expectation is exact.
Remark: Please let us know if you have any remaining concerns or questions. If we have addressed your concerns, we would appreciate it if you could reconsider your score. Thank you again for your constructive feedback.
Thanks for the response. I will maintain my positive score.
Thank you for your acknowledgment and positive evaluation of our work!
We appreciate your valuable comments. We have revised our submission in response to your review, with changes highlighted in red. Below, we provide detailed responses to your questions and concerns.
We added task-order analyses on the original full benchmarks for the three Qwen3.5-9B seeds and extended them to Qwen3.5-9B, Qwen3.5-4B, and Ministral 3 8B, with nine adapter-training seeds per model. Each analysis compares the original order with 20 predetermined permutations; SParC permutations reorder whole dialogues while preserving their internal turns. Candidate pools, priors, learning rates, and comparators remain fixed. Section 7 summarizes the expanded results, with details in Appendix F.2.
The extension uses predefined subsets of 50 Spider questions and 50 complete SParC dialogues in each separate development and held-out partition, selected independently of model outcomes. Development SParC has 160 turns and held-out SParC has 139 turns. Every subset covers all 20 databases, with two or three questions or dialogues per database; each model/seed bank contains 24 prompt–parameter pairs. Cleared-history SParC removes earlier generated SQL while retaining the schema and user utterances. These subset results supplement the completed original full-benchmark analysis.
Full-diagonal exponential weights (EW) observes each candidate's loss on its own history and updates selection probabilities from preceding losses. Follow-the-leader (FTL) chooses the candidate with the lowest cumulative preceding loss; development-fixed selection chooses by development loss and freezes that candidate. The table reports exact expected joint-family held-out failure counts averaged across nine training seeds. Ranges summarize the 20 permuted-order means.
EW has a narrower permutation range than FTL in all six model/benchmark comparisons. FTL changes rank relative to development-fixed selection in 7 of 20 Qwen3.5-9B Spider permutations and 2 of 20 Ministral SParC permutations.
The restricted-feedback replays also show small changes in cumulative loss. Sparse EW probes one candidate per round through an evaluator independent of deployment; deployment-only selection updates from executed losses. Across the six settings, permutation-range widths are 0.011–0.144 failures for sparse independent EW and 0.139–0.269 for deployment-only selection with block length 1. These estimates use 256 selector repetitions per cell. The ranges describe the tested orders and include Monte Carlo variation for stochastic methods.