- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates whether a model whose performance is deliberately reduced behaves like a genuinely weaker model. The authors propose three evaluation dimensions: target score, difficulty profile, and error distribution. They construct CapLadder, covering mathematics, coding, multiple-choice reasoning, and tool use. By training a LoRA adapter using a weaker teacher and adjusting the strength of the learned update, the authors achieve target-score control on some tasks while observing strong answer-letter concentration on multiple-choice questions. Logit interventions and option permutations provide evidence that answer bias contributes substantially to the performance reduction. The paper also examines tool-call error distributions, cross-benchmark calibration transfer, and the limitations of different auditing approaches.
Critical strengths
1. A useful evaluation perspective. Examining target score, difficulty profile, and error distribution separately helps avoid judging whether a model behaves like a weaker model solely from aggregate accuracy. CapLadder provides a common experimental setting for these comparisons.
2. Multiple analyses of the observed anomaly. The authors investigate answer-letter bias using answer logits, shift cancellation, option permutations, and different models and training configurations. They also explicitly limit the scope of the tool-use and transfer findings. These analyses provide more specific evidence about the observed performance reduction than accuracy curves alone.
3. Controllable performance reduction has potential applications, and the approach is worth exploring. Controllably reducing model performance could be useful in role-playing or simulation settings that require different levels of competence. Training a LoRA adapter on a stronger model using a weaker model's outputs, then adjusting the update strength to control its performance, is a simple and promising idea.
Critical weaknesses
1. The basis for the strong–weak reference relationship and behavioral similarity could be strengthened. The paper primarily establishes the relative strength of the base model and weak reference through aggregate accuracy, without sufficiently characterizing the overlap and differences in their item-level correctness. I would like to understand whether items answered correctly by the weaker model are mostly also answered correctly by the base model, and how often the reverse occurs. If there are many items that the weaker model answers correctly but the base model does not, this could indicate different capability profiles, limiting the interpretation of the experiment as reducing the level of the same underlying capability. Human-coded error categories and tool-error distributions provide some behavioral evidence; comparisons of reasoning and tool-selection processes on the same items would further help establish whether the observed similarities extend beyond coarse distributions.
2. The origins of the bias and the scope of its interpretation need further explanation. The extreme answer-letter concentration on multiple-choice questions is a phenomenon worth understanding more deeply. The authors have conducted several diagnostic analyses and controls, so I do not think the existing evidence can simply be characterized as training failure or attributed entirely to extrapolation. However, the respective roles of the training data, supervision, and amplification of the LoRA update remain unclear. Expanding the training details directly relevant to these questions, together with same-item examples from the base model, weak reference, standard LoRA model, and model with an amplified update, would help explain how this behavior develops and assess how far the findings extend beyond the current experimental setting.
Questions for authors
1. The strong–weak model relationship and the representativeness of the experiments. Could the authors provide a cross-tabulation of item-level correctness for the base model and weak reference, showing whether items answered correctly by the weaker model are mostly also answered correctly by the base model, and how often the reverse occurs? This would help assess whether their patterns of competence are approximately nested. In addition, the distillation experiments primarily use models with at most 9B parameters. Please clarify the scope of their implications for stronger models and practical capability evaluations. The prompting experiments with frontier APIs use a different setting from the distillation experiments, and these should be distinguished.
2. Similarity in reasoning and tool-selection patterns. The authors already compare human-coded error categories and tool-error distributions. Same-item examples of complete generations, comparing the base model, weak reference, and throttled model in terms of reasoning steps, tool selection, and argument construction, would help explain whether behavioral similarity extends beyond final answers and coarse error distributions.
3. How answer bias develops. The authors provide several diagnostics and controls for answer-letter bias, including results showing collapse without update amplification under other training configurations. For the main SFT method, I would still appreciate a clearer account of the behavioral differences between standard LoRA () and amplified updates, and the respective contributions of training and subsequent scaling. Including both settings in the same-item examples discussed above would also help illustrate how the bias manifests during generation.
4. The composition and supervision of the final training data. Table 9 reports the teacher's answer-letter distribution, but does this correspond to the actual training targets after filtering and processing? If the training data are filtered, how does filtering change their composition, and what are the final proportions of different sample types? A brief explanation of how teacher completions are supervised, and of the relationship between the distillation training set, calibration set, and test set, would help interpret the behavioral changes induced by training. Training-data filtering should be distinguished from item filtering for IRT difficulty analysis; the latter cannot automatically be assumed to describe the training-data pipeline.
5. The scope of the IRT difficulty ranking and the effects of filtering. Please clarify which models are evaluated using the difficulty ranking fitted solely from Qwen responses. If it is used for other model families, evidence supporting its applicability would be helpful. Within the same family, is the ranking stable when the pool of models used for fitting changes? The original task composition is reported in Table 6, but the proportion of items retained after IRT fitting differs across tasks. It would be useful to understand the characteristics of excluded items and whether this filtering affects the conclusions drawn from difficulty-profile comparisons.
6. Interpretation of the relative-distractor metric . Cyclic differences between option positions have no inherent semantic meaning. A model that always selects D can also produce a uniform relative-position distribution under certain answer-key distributions. Please explain what this metric is intended to identify and how it supports judgments about similarity in error patterns. This question concerns the interpretation of and does not dispute the answer-letter bias revealed by .
7. Consistency in calibration-error reporting. Equation (4) defines the target using the test-set base accuracy, whereas Appendix C states that targets are derived from the calibration set. Using the test-set base accuracy of 0.902 and the achieved accuracies of 0.722 and 0.559 for the 80% and 60% targets, Equation (4) gives a mean error of approximately 0.9 percentage points. Using the targets of 0.711 and 0.533 in Table 8 instead gives approximately 1.85 percentage points. In addition, the Table 2 caption specifies an averaging grid of 80%/60%/40%, whereas Appendix C specifies 80%/60%. Please clarify the definition, target grid, and calculation used for the reported 1.9-percentage-point error.
8. Interpretation of the answer-key floor. Equation (6) bounds expected accuracy when responses are independent of the gold answers. Does interpreting a score below this range as using knowledge of the correct answer to avoid it also require ruling out explanations such as systematic task misunderstanding or an incorrect mapping between answers and labels? Representative outputs explaining this behavior, together with clarification of the criterion's statistical meaning for finite samples, would make the conclusion clearer.
9. Clarifications on figures and result reporting. Please explain why the Open 4B results for the 80% and 60% conditions in Table 2 are reported as “≈0.90,” and why Gemini's mean target miss is described as “cliff” rather than reported using the same metric as the other columns. In Figure 1B, the weak reference's maximum letter share is labeled as 0.22, although the largest plotted value is 0.27; please check this annotation. The upper and lower parts of Figure 1C use different LoRA scaling strengths; labeling them separately would also help readers understand how the results relate to one another.