- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies layer skipping, repetition, and substitution in frozen language models. It measures additional answer coverage, the ability of selectors to recover those answers, and how combined edits behave across prompts.
Critical strengths
- The evaluation tracks candidate availability, shortlist retention, and final answers. It shows where additional correct candidates are lost during selection, providing useful evidence about the gap between coverage and output accuracy.
- The composition analysis identifies combinations that succeed even when both component edits fail. About 45–51% of these discovered cases succeed again under a rewritten prompt, revealing useful limits of predicting combinations from their components.
Critical weaknesses
- Correctness labels need stronger validation. The paper documents a checker false positive, grading disagreements, and incomplete generation records. A single falsely accepted answer can inflate coverage across 1,260 edits, making independent verification important for the central finding.
- The coverage comparison uses 1,260 edited paths and 57 ordinary samples, leaving the contribution of candidate count unresolved. The 57-path voter also answers 23–57 fewer questions correctly than self-consistency across the three groups. Missing runtime and token measurements leave its practical tradeoff unclear.
Questions for authors
- Could you independently verify the additional-coverage cases and the cases determining the voting comparison, using consistent grading and confirmed question and path identities?
- Could you compare coverage at shared candidate counts and report generation time, output tokens, and feature-extraction costs for the 57-path voter and self-consistency?