- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper notes that large LLMs are really good but expensive in terms of cost, memory, and usage. On the contrary, small models are cheaper and can run locally, but they perform worse than larger models, and adaptation can narrow that gap for a particular task. To this end, the paper’s practical question is which adaptation method to use for a particular task and model. Five adaptation methods are evaluated: ICL (in-context learning), SkillOpt, SFT (supervised fine-tuning), RL (reinforcement learning), and TTT (test-time training). The first two leave the model weights untouched, while the last three update them. The authors test Qwen3-8B and Qwen 27B targets on 16 diverse benchmarks, with Kimi K3 and DeepSeek v4 Pro as large-model references. They find that the best method depends on the task and target model, identify error repairability, workload composition, and supervision availability as possible clues, and compare adaptation costs. Based on this, they suggest a recipe for a new task. First, run the model on a few examples and profile the errors while knowing how much training data exists. Use the results from similar past tasks to decide which method to try first, within the budget. Try the methods and pick the one with the best validation score. Lock in the choice before touching the test set. Only this final selection step is evaluated on four additional benchmarks, where it picks correctly in some cases and makes ranking errors in others.
Critical strengths
-
The paper presents a holistic and broad comparison that includes five adaptation methods, comparing on 16 benchmarks across 6 task categories.
-
The paper’s value is a guide for budding practitioners, and its breadth is what produced findings and claims that a narrow study could not produce.
-
The paper has concrete, actionable diagnostics and cost findings; these give a reader something to act on.
Critical weaknesses
-
The experiments are usually single runs, and the methods are served on different backends (ICL and SkillOpt on OpenRouter, SFT and RL on Tinker), whose unadapted controls differ by up to about 17 points (Table 3). The observed method ranking could come from these serving setup and protocol differences rather than the methods themselves. Since the contribution is purely empirical, this undercuts the core of the paper.
-
The guidance is never fully validated in the paper. Only the last step is evaluated, and that step simply picks the method with the highest validation score, which is standard practice. I am also concerned that, as the paper itself admits, the evaluation doesn’t measure its exploration efficiency or the benefit of diagnostic retrieval.
-
Large models appear only as unadapted quality references. The paper never compares the cost of adaptation with simply using the large model.
-
No target model families beyond Qwen are reported, and no wider range of sizes than 8B and 27B as well.
Questions for authors
-
Can you compare the full cost of an adapted small model against simply using a large model at the same required quality?
-
Qwen 27B RL uses Qwen3.8, while the other 27B methods use Qwen3.6. Does the 27B method ordering change if you restrict it to a single version?
-
For a subset of tasks, rerun ICL, SkillOpt, SFT and RL on one backend, with one model version and at least three seeds, so the method comparison isn’t confounded.