- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper studies the setting of a larger model decomposing tasks for smaller models. In particular, the work is interested in whether smaller, instruction-tuned models can complete tasks given strategies and partial work from larger models and whether they can detect errors inserted into these strategies or partial work. They find that most instruction-tuned models are able to follow the follow provided strategies and continue partially complete work, but they are generally not able to identify and correct errors. Finally, the work uses interpretability techniques to illustrate the source of these issues.
Critical strengths
- As models are increasingly used as parts of multi-agent systems, questions around how and whether to delegate work to weaker models leads to interesting research problems like the ones studied in this work.
- The work studies a number of different task settings along with a number of different models, illustrating that their findings are not model-specific.
- Finally, the finding that in the cases where models report errors (lines 350-352), they re-derive answers is neat.
Critical weaknesses
- A critical weakness of the work was the choice to focus on instruction models rather than reasoning models. For the reasoning-intensive tasks that are studied in this work, reasoning models are more appropriate to study compared to instruction-tuned models. It makes more sense to provide reasoning models with reasoning strategies or partial reasoning traces and see if they can detect and correct errors. The one reasoning model studied performs well, but the discussion seems like it is cut short.
- Additionally, while it is appreciated that open-weight models are the focus of this work, if possible it would be good to include frontier models in the analysis as well. These are models that would likely be used by practitioners, so it is worth having a sense of how they perform.
- Finally, while mentioned by the authors, putting the partial solution into the assistant message is likely not the most ecologically valid design choice—-during training, the assistant message is only generated by the model, so injecting something into it does not necessarily give a natural response.
Questions for authors
not questions, just flagging some typos:
- line 70: “at at”
- line 129: “trough” → “through”
- line 198/199: “you” is used in the first sentence but “we” is used in the second.
- line 201: “which” should be capitalized
Thanks for your detailed responses. I have raised my score.