acceptodds
Withdrawn from ICLR 2027

Can Adaptation Substitute for Model Scale? An Empirical Study of Language Model Capability Gaps

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Smaller language models offer a practical deployment option, and a range of adaptation methods can narrow their performance gaps relative to larger models. However, choosing a method for a particular workload requires more than knowing its best attainable score. We present an empirical study across 16 generalization and discovery benchmarks, 8B and 27B targets, and five strategies spanning prompting, harness optimization, and weight updates. We compare method outcomes and score margins, separating heterogeneous endpoint inventories from gains against corresponding unadapted controls. The evaluated methods exhibit both large performance differences and near ties, and their ordering can change with the target model. Behavioral analyses reveal benefits concentrated in particular task strata and changes accompanying reduced output failures. Cost decomposition relates method choice to quality requirements, adaptation expenditure, and reusable supervision. We propose an experience-guided mechanism that prioritizes feasible probes under a budget and selects by target validation. Replay of its validation gate on four additional benchmarks reveals both successful choices and ranking errors. Code and experimental results are available at https://anonymous.4open.science/r/adaptation-study-57D6/.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Small Language Models, Capability Gaps, Model Adaptation, Workload-Aware Method Selection, Cost–Performance Trade-offs, Empirical Evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_scFoOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

The paper notes that large LLMs are really good but expensive in terms of cost, memory, and usage. On the contrary, small models are cheaper and can run locally, but they perform worse than larger models, and adaptation can narrow that gap for a particular task. To this end, the paper’s practical question is which adaptation method to use for a particular task and model. Five adaptation methods are evaluated: ICL (in-context learning), SkillOpt, SFT (supervised fine-tuning), RL (reinforcement learning), and TTT (test-time training). The first two leave the model weights untouched, while the last three update them. The authors test Qwen3-8B and Qwen 27B targets on 16 diverse benchmarks, with Kimi K3 and DeepSeek v4 Pro as large-model references. They find that the best method depends on the task and target model, identify error repairability, workload composition, and supervision availability as possible clues, and compare adaptation costs. Based on this, they suggest a recipe for a new task. First, run the model on a few examples and profile the errors while knowing how much training data exists. Use the results from similar past tasks to decide which method to try first, within the budget. Try the methods and pick the one with the best validation score. Lock in the choice before touching the test set. Only this final selection step is evaluated on four additional benchmarks, where it picks correctly in some cases and makes ranking errors in others.

Critical strengths
  1. The paper presents a holistic and broad comparison that includes five adaptation methods, comparing on 16 benchmarks across 6 task categories.

  2. The paper’s value is a guide for budding practitioners, and its breadth is what produced findings and claims that a narrow study could not produce.

  3. The paper has concrete, actionable diagnostics and cost findings; these give a reader something to act on.

Critical weaknesses
  1. The experiments are usually single runs, and the methods are served on different backends (ICL and SkillOpt on OpenRouter, SFT and RL on Tinker), whose unadapted controls differ by up to about 17 points (Table 3). The observed method ranking could come from these serving setup and protocol differences rather than the methods themselves. Since the contribution is purely empirical, this undercuts the core of the paper.

  2. The guidance is never fully validated in the paper. Only the last step is evaluated, and that step simply picks the method with the highest validation score, which is standard practice. I am also concerned that, as the paper itself admits, the evaluation doesn’t measure its exploration efficiency or the benefit of diagnostic retrieval.

  3. Large models appear only as unadapted quality references. The paper never compares the cost of adaptation with simply using the large model.

  4. No target model families beyond Qwen are reported, and no wider range of sizes than 8B and 27B as well.

Questions for authors
  1. Can you compare the full cost of an adapted small model against simply using a large model at the same required quality?

  2. Qwen 27B RL uses Qwen3.8, while the other 27B methods use Qwen3.6. Does the 27B method ordering change if you restrict it to a single version?

  3. For a subset of tasks, rerun ICL, SkillOpt, SFT and RL on one backend, with one model version and at least three seeds, so the method comparison isn’t confounded.

Reviewer_AFgFOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how different adaptation strategies affect smaller language models across workloads. It compares few-shot prompting, harness optimization, supervised fine-tuning, reinforcement learning, and test-time training across 16 benchmarks and 8B/27B targets, with larger models serving as unadapted references. The analysis examines method rankings, task-specific error patterns, supervision availability, and adaptation costs. The paper also proposes an experience-guided mechnasim and evaluates its final validation-based selection gate on four additional benchmarks.

Critical strengths
  1. The study addresses a useful practical question: "when should a practitioner invest in prompting, harness optimization, or weight updates?" Comparing these interventions across diverse workloads is valuable, particularly because the results show that simple methods can remain competitive and that expensive adaptation is not uniformly beneficial.

  2. The paper is unusually explicit about the limits of its evidence. It documents model and evaluation differences, provides same-backend controls for SFT, separates adaptation expenditure from final evaluation, and distinguishes post-hoc diagnostics from held-out results. The FinQA output-convention analysis is also informative: it shows how a substantial apparent performance gap can reflect a repairable execution issue rather than a lack of underlying reasoning ability.

Critical weaknesses
  1. The practical lessons for a new workload remain somewhat unclear. The study provides a useful inventory of adaptation outcomes and carefully documents differences between recipes. However, observations (shown as Figure 3) such as the different method preferences for 8B and 27B targets are difficult to translate into guidance beyond the evaluated settings. The diagnostics offer possible explanations, but are largely descriptive or post hoc. A clearer distinction between findings that practitioners can reasonably use as starting points and findings that still require target-specific validation would strengthen the study.

  2. The experience-guided selection policy is not evaluated as a complete policy. The four additional benchmarks test a validation-based selection gate over an existing candidate library, while bypassing the history-based probe ordering and budget stopping rule, as explicitly acknowledged in Appendix E. These results assess the final selection step, but do not establish whether the proposed diagnostics or historical similarity reduce the cost of finding a good adaptation method. This limits the evidence for the selection mechanism, although it does not invalidate the preceding empirical observations.

Questions for authors
  1. Which findings would you recommend practitioners use to decide what to try first on a new workload? In particular, how broadly should the observed difference in method preference between 8B and 27B targets be interpreted, given the documented recipe and model differences?

  2. Do the proposed diagnostics provide useful guidance beyond simple task categories or a fixed method preference? Any existing evidence showing when they correctly anticipate an adaptation outcome and when they fail would help clarify their practical value.

  3. Is there evidence that the history-based probe ordering finds a satisfactory method at lower exploration cost than random or cheapest-first ordering? If this has not been evaluated, I would suggest presenting the sequential policy as a proposal and distinguishing it more clearly from the validated selection gate.

Reviewer_fXruOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper compares a wide range of different adaptation methods (in-context learning, prompt optimization, SFT, RL, and test-time-training) on a variety of tasks and across three large language models.

Critical strengths

The paper presents a wide empirical study. The paper uses a clear methodology for separating training, validation, and test data, and builds a reasonable framework for accounting for adaptation cost.

Critical weaknesses

Given the central question in the paper title, "Can adaptation substitute for model scale?", I would expect the paper to frame its experiments around this question, and give a clear and compelling answer to this question. However, this is not the case. Furthermore, since the paper's study is limited to two parameter scales (8B and 27B), it is difficult to draw a confident conclusion to this answer.

Since this is an analysis paper and understandably there is little novelty in terms of method, the value of the paper is largely determined by the "interestingness" and impact of its results. In my opinion, these results are quite mixed and there is particularly clear takeaway. For example, I don't find any of the three insights in Section 3 compelling. This sentiment extends to Section 4, where the notions of "error repairability" (which has a confusing definition) and "workload composition" do not generate takeaways, and instead paint a complicated picture. When it comes to analysis of teacher quality and supervision examples, there is plenty of related work in distillation and ICL (mostly not cited) which does a better job here.

The empirical results in Figure 3 are incomplete, and sometimes draw comparisons between different base models (I am glad this is clearly annotated with the hatching, but it is a very confusing choice of method) and is suggestive of other deficiencies in the experimental setup.

The main novelty in terms of method, the "experienced-guided selection mechanism" introduced in Section 6 is more of an afterthought, and in my view the practical use for this particular meta-selection algorithm is limited. The evaluation in the paper of this algorithm is also limited to only 4 new benchmarks, without any comparison to reasonable baselines.

Questions for authors

Why is TTT not applied to the first set of benchmarks in Figure 3, and why is RL and SFT not applied to other rows? In particular, why are there protocol differences and what is the difference between unassigned missing and excluded?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.