- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
FINALE turns successful teacher trajectories for web/game development (39K trajectories from Claude Opus 4.7) into extra "local" training units. ArtifactReplay groups the edit history of files that survive into the delivered project, and BuildReplay takes edit phases that end in a passing build. For each unit, FINALE rolls back the repository to its prior state, has an LLM write a local request, and supervises only the endpoint-supported teacher actions (writes containing at least one change that persists to the endpoint, their supporting Read/Locate calls, and passing builds). Mixed with the full trajectories for fine-tuning, it lifts Qwen3.5-35B-A3B-Base to 47.29 and Qwen3.6-27B to 62.22 on ArtifactsBench and Cookie-Bench, which the paper says exceeds DeepSeek-V4-Pro.
Critical strengths
- A practical design: using retained file states and passing builds as observable endpoints gives verifiable segment boundaries.
- The construction is careful: state-recovery rules, replay checks, and supervision masks are clearly specified and deterministically validated.
- The paper includes several useful controls (step-matched training, intermediate replay, reweighting) and reports behavior changes such as fewer repeated writes.
Critical weaknesses
-
The gain is not cleanly separated from extra training. FINALE processes about 2.4× the tokens of the two-epoch full-trajectory baseline for its 4.8-point gain. Both longer-schedule controls (over-repeated full trajectories and Matched Intermediate Replay, which roughly matches FINALE's token exposure) score below that baseline, so extra exposure alone does not obviously explain the gain. But the abstract's "comparable token budget" claim rests on a degraded baseline, and FINALE is never run at the full-trajectory baseline's budget, which is the comparison that would settle it.
-
The surprising component interaction rests on one training seed. In the reported run, adding ArtifactReplay, the vast majority of local units, lowers the score below full-trajectory training (Full+A 64.0 vs. 67.9), yet adding the small BuildReplay set as well makes the combination the best arm. The "multiple seeds" in Appendix F.4 only resample evaluation tasks, so training-seed robustness is unknown.
-
The claimed mechanism, local request plus recovered prestate, is not tested directly. Same-Target Reweighting keeps the global prompt, but it also drops the local units and trains for 283 rather than 682 steps, so it does not separate conditioning from data volume. No run swaps in the global query or removes the prestate while keeping everything else fixed. QueryLM also sees the target edits, so a blinded human audit of sampled local units is needed to check that the requests are faithful, the file groups are coherent, and the requests do not leak the answer.
-
The headline comparison does not isolate FINALE from ordinary distillation, and judge dependence is untested. Table 1 has no full-trajectory baseline, so much of the gain over the base models may come from fine-tuning on Opus 4.7 trajectories; on the ablation subset, full-trajectory training alone gives most of FINALE's improvement. The cross-model ranking also rests on a single judge (Claude Sonnet 4.6, from the teacher's family), so re-judging the Table 1 outputs with an independent judge is needed.
-
The claim of beating DeepSeek-V4-Pro depends on weighting each subset equally. The published number is a macro-average over subsets of different size; weighting by task count gives 59.35 for FINALE vs. 60.81 for DeepSeek-V4-Pro, which reverses the claim.
-
Overlap between training requests and benchmarks cannot be ruled out. The paper does not report where the 39K training requests come from, so exact and semantic overlap checks against both benchmarks are needed.
-
Results are tied to an undisclosed in-house scaffold. Agent harnesses strongly affect end-to-end scores, yet the teacher scaffold is described in one line and the evaluation harness for FINALE and the baselines is not described at all. If FINALE is evaluated in the scaffold it was trained on while baselines such as DeepSeek-V4-Pro are run in an unfamiliar one, Table 1 conflates scaffold fit with model quality. Evaluating all models in a widely used open-source agent (e.g., OpenCode or a similar public harness), or at least releasing the scaffold's tools and prompts, would make the comparison fair and reproducible.
-
Limited conceptual novelty. The core idea, relabeling segments of successful trajectories by the outcomes they actually achieved and supervising only the actions that contributed, is hindsight relabeling (HER) applied to coding-agent SFT, and close to recent LLM-based variants such as HSL. The contribution is mainly the choice of anchors (retained file state, passing builds) and careful replay engineering. That can be valuable, but then the paper needs to show that these anchors beat LLM-inferred boundaries, and no LLM-segmentation baseline such as HSL is run.
Questions for authors
- How does FINALE compare with full-trajectory training when run at the same token budget?
- What is the run-to-run variance of the main ablation arms over several training seeds?
- What happens with the global query instead of the local one, or without the prestate, with everything else fixed?
- What is the full-trajectory baseline's score in Table 1, and do the rankings hold under an independent judge?
- Where do the training requests come from, and how much do they overlap with ArtifactsBench and Cookie-Bench prompts?
- Which harness was used to evaluate each model in Table 1, and do the results hold in a public open-source agent scaffold?
- How is a "successful" teacher trajectory defined (build pass, judge score, or other), and what fraction of collected trajectories was discarded?