- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper fine-tunes Qwen3.6-35B-A3B to manage long contexts through a minimal harness with two tools, read_chunk and spawn_subagent. Each agent works under a small context budget, 8K tokens at evaluation. LoRA SFT on about 1.2K synthetic problems teaches three strategies: tree reduction, sequential folding and boundary handling. The model is trained only on 4–14K-token documents, yet it stays at or above 0.85 on RULER up to 320K tokens. On OOLONG-synth it is reported to match GPT-5.4 at long lengths. An RL stage was attempted without success.
Critical strengths
S1. The formulation is clean and minimal. Internalizing context decomposition through training, rather than hard-coding it into a harness, is a timely idea. The two-tool harness is simple and fully specified, which makes the setup easy to understand.
S2. The length-generalization result is notable and honestly reported. Although the model is trained on documents of at most 14K tokens, it operates on inputs of up to 320K tokens, and RULER NIAH stays at 1.00 throughout. Negative results are also reported candidly, including the CWE collapse and the unsuccessful RL stage.
Critical weaknesses
W1. The evaluation is too small, and the main claim is stated inconsistently. RULER uses 5 questions per task type and OOLONG-synth uses 30 problems per length. All results come from single runs without confidence intervals. At n = 30 the uncertainty is roughly ±15–18 pp, which is too large to support a claim of parity. The claim itself also varies: the abstract states parity with GPT-5.4 beyond 40K tokens, whereas Sec. 1.2 and Sec. 4 say beyond 80K. Reading approximately from Fig. 4, the model still trails GPT-5.4 at both 40K and 160K.
W2. Key baselines that would isolate the effect of training are missing. The paper reports neither base Qwen in the same harness nor a scripted harness that implements the same tree and fold strategies with base Qwen at the leaves. It also omits existing methods such as RLM and MemAgent, as well as base Qwen at its native context length beyond the 64K cap imposed by Tinker. Without the scripted-harness comparison in particular, it is unclear whether the gains come from training or from the hand-designed decomposition strategies encoded in the data.
W3. The training and evaluation distributions overlap substantially. The conclusion states that the authors "do not train on our evaluation distributions" (l. 354), yet several training tasks closely mirror the evaluation tasks: vt_novel resembles RULER VT, niah_* resembles NIAH, synth_topk resembles CWE/FWE, and labeled_records mirrors the construction of OOLONG-synth. Please confirm that the seven labelled datasets do not overlap with OOLONG-synth's sources, and evaluate on tasks with genuinely different structure.
W4. Data generation and inference cost are under-specified. It is unclear how the demonstrations were generated and verified, and how many training examples and tokens were used. Inference cost (number of subagents, total tokens, latency) and failure rates are not reported, which makes a fair comparison with a single GPT-5.4 call difficult.
W5. The study lacks depth, and the extent of automated research is unclear. The main text is about seven pages and contains relatively little experimental analysis, and the RL stage is described only anecdotally. According to the AI-use statement, generative AI tools were used at nearly every stage of the work. This includes writing all code, running experiments, creating figures, generating data, proposing hypotheses, designing the methodology and interpreting results. I appreciate this candid disclosure, and such use is permitted. Together with the limited depth of analysis, however, it gives the impression that the work may largely be the output of an automated research pipeline. A clearer account of how the authors verified the experiments, data and results, together with a deeper analysis of the learned behaviour, would increase confidence in the findings.
Questions for authors
Please see the Weaknesses section.