acceptodds
Under review as a conference paper at ICLR 2027

Finale: Reconstructing Local Supervision with Terminal-Anchored Replay for End-to-End Web Agents

Abstract

End-to-end Web development requires implementing and integrating interdependent features. Complete teacher trajectories demonstrate both local implementation and the overall development process. We investigate whether more explicit local supervision from existing teacher demonstrations can improve end-to-end performance beyond complete-trajectory training alone. Identifying these local tasks is difficult because repeated file revisions leave multiple candidate endpoints, while local objectives and action boundaries remain unlabeled. We introduce \method, which reconstructs these tasks using retained file states and successful builds as observable endpoint anchors. It comprises two reconstruction workflows: ArtifactReplay groups file histories contributing to shared functionality retained in the delivered project, while BuildReplay extracts edit phases ending in successful builds. For each unit, \method recovers the repository prestate, synthesizes a local query from recorded target evidence, and uses endpoint-supported teacher actions as supervision for the model. We mix these units with complete trajectories for training. Across the Web and Game sets of ArtifactsBench and Cookie-Bench, \method improves Qwen3.5-35B-A3B-Base to an average score of 47.29, surpassing its official post-trained counterpart by 17.2%. Applied to Qwen3.6-27B, it raises the score from 40.73 to 62.22, exceeding DeepSeek-V4-Pro (59.05). In controlled comparisons, \method outperforms complete-trajectory training by 4.8 points under the same two-epoch schedule and by 17.2 points at comparable processed-token budgets, while surpassing intermediate replay and same-target reweighting.

Primary area:
foundation or frontier models, including LLMs
Keywords:
agentic LLM, LLM, post training, data

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_uM5hOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

FINALE turns successful teacher trajectories for web/game development (39K trajectories from Claude Opus 4.7) into extra "local" training units. ArtifactReplay groups the edit history of files that survive into the delivered project, and BuildReplay takes edit phases that end in a passing build. For each unit, FINALE rolls back the repository to its prior state, has an LLM write a local request, and supervises only the endpoint-supported teacher actions (writes containing at least one change that persists to the endpoint, their supporting Read/Locate calls, and passing builds). Mixed with the full trajectories for fine-tuning, it lifts Qwen3.5-35B-A3B-Base to 47.29 and Qwen3.6-27B to 62.22 on ArtifactsBench and Cookie-Bench, which the paper says exceeds DeepSeek-V4-Pro.

Critical strengths
  • A practical design: using retained file states and passing builds as observable endpoints gives verifiable segment boundaries.
  • The construction is careful: state-recovery rules, replay checks, and supervision masks are clearly specified and deterministically validated.
  • The paper includes several useful controls (step-matched training, intermediate replay, reweighting) and reports behavior changes such as fewer repeated writes.
Critical weaknesses
  1. The gain is not cleanly separated from extra training. FINALE processes about 2.4× the tokens of the two-epoch full-trajectory baseline for its 4.8-point gain. Both longer-schedule controls (over-repeated full trajectories and Matched Intermediate Replay, which roughly matches FINALE's token exposure) score below that baseline, so extra exposure alone does not obviously explain the gain. But the abstract's "comparable token budget" claim rests on a degraded baseline, and FINALE is never run at the full-trajectory baseline's budget, which is the comparison that would settle it.

  2. The surprising component interaction rests on one training seed. In the reported run, adding ArtifactReplay, the vast majority of local units, lowers the score below full-trajectory training (Full+A 64.0 vs. 67.9), yet adding the small BuildReplay set as well makes the combination the best arm. The "multiple seeds" in Appendix F.4 only resample evaluation tasks, so training-seed robustness is unknown.

  3. The claimed mechanism, local request plus recovered prestate, is not tested directly. Same-Target Reweighting keeps the global prompt, but it also drops the local units and trains for 283 rather than 682 steps, so it does not separate conditioning from data volume. No run swaps in the global query or removes the prestate while keeping everything else fixed. QueryLM also sees the target edits, so a blinded human audit of sampled local units is needed to check that the requests are faithful, the file groups are coherent, and the requests do not leak the answer.

  4. The headline comparison does not isolate FINALE from ordinary distillation, and judge dependence is untested. Table 1 has no full-trajectory baseline, so much of the gain over the base models may come from fine-tuning on Opus 4.7 trajectories; on the ablation subset, full-trajectory training alone gives most of FINALE's improvement. The cross-model ranking also rests on a single judge (Claude Sonnet 4.6, from the teacher's family), so re-judging the Table 1 outputs with an independent judge is needed.

  5. The claim of beating DeepSeek-V4-Pro depends on weighting each subset equally. The published number is a macro-average over subsets of different size; weighting by task count gives 59.35 for FINALE vs. 60.81 for DeepSeek-V4-Pro, which reverses the claim.

  6. Overlap between training requests and benchmarks cannot be ruled out. The paper does not report where the 39K training requests come from, so exact and semantic overlap checks against both benchmarks are needed.

  7. Results are tied to an undisclosed in-house scaffold. Agent harnesses strongly affect end-to-end scores, yet the teacher scaffold is described in one line and the evaluation harness for FINALE and the baselines is not described at all. If FINALE is evaluated in the scaffold it was trained on while baselines such as DeepSeek-V4-Pro are run in an unfamiliar one, Table 1 conflates scaffold fit with model quality. Evaluating all models in a widely used open-source agent (e.g., OpenCode or a similar public harness), or at least releasing the scaffold's tools and prompts, would make the comparison fair and reproducible.

  8. Limited conceptual novelty. The core idea, relabeling segments of successful trajectories by the outcomes they actually achieved and supervising only the actions that contributed, is hindsight relabeling (HER) applied to coding-agent SFT, and close to recent LLM-based variants such as HSL. The contribution is mainly the choice of anchors (retained file state, passing builds) and careful replay engineering. That can be valuable, but then the paper needs to show that these anchors beat LLM-inferred boundaries, and no LLM-segmentation baseline such as HSL is run.

Questions for authors
  1. How does FINALE compare with full-trajectory training when run at the same token budget?
  2. What is the run-to-run variance of the main ablation arms over several training seeds?
  3. What happens with the global query instead of the local one, or without the prestate, with everything else fixed?
  4. What is the full-trajectory baseline's score in Table 1, and do the rankings hold under an independent judge?
  5. Where do the training requests come from, and how much do they overlap with ArtifactsBench and Cookie-Bench prompts?
  6. Which harness was used to evaluate each model in Table 1, and do the results hold in a public open-source agent scaffold?
  7. How is a "successful" teacher trajectory defined (build pass, judge score, or other), and what fraction of collected trajectories was discarded?
Reviewer_JsQ4OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Finale turns complete web-development trajectories into local training tasks. ArtifactReplay anchors tasks to retained file contents, while BuildReplay extracts edit phases ending in successful builds. The method reconstructs repository prestates, generates local requests, and selectively supervises recorded teacher actions. Mixing these units with complete trajectories improves two Qwen checkpoints. Controlled experiments show a 4.8-point gain over two-epoch full-trajectory training and improvements over intermediate replay and target reweighting.

Critical strengths

The paper addresses a useful training problem: a long demonstration contains local implementation work, but its global request does not explicitly describe each local objective. Combining observable endpoints with recovered repository states is a concrete way to make that supervision more explicit. The contribution is not hindsight relabeling alone, but its implementation for stateful, multi-file development trajectories.

The reconstruction procedure is technically careful. It distinguishes unknown from absent files, preserves tool-call/observation pairing, rejects ambiguous state recovery, and verifies endpoint equality through deterministic file replay. The appendices specify the masking, grouping, and query-generation rules in substantial detail.

The empirical study goes beyond comparing against an untuned checkpoint. It includes full-trajectory baselines, a processed-token-matched control, intermediate replay, target reweighting, and component ablations.

Critical weaknesses

1. The headline gains need a more careful interpretation of training exposure. The 4.8-point comparison uses the same number of epochs, but Finale processes 4.247B tokens versus 1.752B for Full-only. The 17.2-point comparison approximately matches processed tokens, but Full-only receives about 4.82 passes over its data and declines from 67.9 to 55.5. Its positive-supervision exposure also differs substantially: 3.023B tokens versus Finale's 1.934B.

The authors disclose these differences and appropriately avoid claiming that overtraining is established. Nevertheless, the large compute-matched margin partly reflects a weakened baseline under the longer schedule. A training-budget curve with validation-selected checkpoints or learning-rate sensitivity would make the efficiency claim more persuasive. The matched intermediate-replay result is a useful complementary control, but it does not replace a well-tuned full-trajectory baseline.

The component results also deserve explanation: Full+A scores 64.0, below Full-only's 67.9, whereas Full+A+B reaches 72.7. Since the schedules differ, this does not cleanly establish complementary effects of the two replay types.

2. The evaluation only partially establishes improved application functionality. The main results use Claude-Sonnet-4.6 on the full benchmark sets, while controlled comparisons use Qwen2.5-VL-72B on 200-task subsets. These tracks are explicitly distinguished, so their different score scales are not an inconsistency. However, the Full-only comparison is not shown under the main evaluation track or for the 27B backbone. The large gains over original checkpoints therefore combine the effects of teacher-trajectory training and the proposed local replay.

Cookie-Bench uses the static evaluator rather than its interaction-based evaluation. ArtifactsBench includes scripted interaction, but both benchmarks ultimately rely on model judgments. The manual examples are informative, not a representative functional audit.

3. File-state validity does not establish the semantic quality of reconstructed tasks. TerminalMatch marks an entire write positive when at least one substantive addition or deletion survives. A large file-creation call can therefore receive supervision even if much of its content is later replaced. Likewise, deterministic grouping constraints establish coverage and non-containment, not that a group corresponds to one complete functional objective.

The 500-example analysis measures task diversity, not task quality. Assigning labels such as "UI" or "core logic" does not establish whether a generated request accurately describes the recorded edits, whether the grouped files implement one shared feature, or whether the training targets still contain substantial code that the teacher later replaced. A small manual audit should check these properties directly.

The ablations also leave unclear which part of Finale produces the improvement. The authors should remove the generated local request, restored repository context, and terminal-based action selection one at a time, while holding the underlying data and training budget fixed. Same-Target Reweighting does not isolate these effects: it changes the input context and loss weighting together, using only one weight setting. A matched hindsight-relabeling baseline would further test whether terminal anchoring adds value beyond simply generating local goals from completed trajectories.

Questions for authors
  1. Does Finale retain its advantage over a validation-tuned Full-only baseline under matched computation, including under the main judge and on the 27B backbone?
  2. How do the results change when local queries, recovered prestates, and terminal masking are ablated separately? What does a human audit find about local-task coherence?
  3. What are the sources of the training requests, the trajectory-success criterion, and the procedures for excluding benchmark overlap?
Reviewer_TPJHOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

The paper asks whether end-to-end Web-development SFT improves when local implementation objectives are made explicit, rather than training only on complete teacher trajectories. Motivated by evidence that agents often rewrite files they have already produced, leaving multiple candidate endpoints with no local labels, it proposes FINALE. FINALE treats two observable terminals, the content retained at delivery and the state after a passing build, as anchors and reconstructs local supervision units around them: the writes that survive to the endpoint, the reads they depend on and the closing build are recovered by backward inversion of the trajectory, and the units are trained with a masked negative log-likelihood objective. FINALE significantly improves average judged quality over the base models and over the compared post-trained models.

Critical strengths
  1. The paper is well written and easy to follow.

  2. The reported gains are substantial. FINALE improves average judged quality over the base models and over the compared post-trained models across benchmarks, and the behavioral measures move in the same direction: fewer steps, more files touched and higher build completion.

  3. The data construction is anchored to observable terminal states rather than to an LLM's opinion about where a step begins or ends: a unit is kept only if the corresponding write survives to delivery or to a passing build. That makes the released dataset rebuildable and auditable by someone else, which is rare for local-supervision data

Critical weaknesses
  1. The comparison that matters most is missing. Table 1 has no full-trajectory SFT baseline, so it does not show how much FINALE gains over training on complete trajectories under the headline setting. The only full-trajectory control is in Table 2, which uses a different judge and a smaller 200-query subset, so its result cannot be carried back to Table 1. Table 1 therefore shows that FINALE beats the base and post-trained models, but not the comparison the paper's claim depends on.

  2. The local query can leak the solution. QueryLM is conditioned on the target file paths, the target messages and the recovered prestate, and the student only sees the generated query. The leakage gates reject exact target strings, long token overlaps and target-only identifiers, and the paper states that this does not establish semantic non-leakage. A query that paraphrases the target change makes the unit easier than a real user request and can inflate the measured gain; it also means the local objective is derived from the answer rather than independently labeled.

  3. In Table 2, Full+A scores 64.0, below Full-only at 67.9, even though it trains for 640 steps instead of 283, so adding A alone hurts on all four scenarios. Adding B then raises the average to 72.7, and the paper concludes that both components are needed. But Full+B trains for 325 steps and FINALE for 682, so the +3.7 over Full+B includes more than twice the optimization. In the near-matched comparison against Full+A (682 vs 640 steps), B's own positive-token share is 0.35% in Full+B, yet the average rises by 8.7 points, which is an interaction between mixture ratios rather than an additive contribution. The ablation shows that the mixture works, not that each component contributes as claimed.

Questions for authors
  1. Table 1 and Table 2 score the same FINALE checkpoint 50.38 and 78.0 on ArtifactsBench Game. How much of this gap comes from the judge and how much from the 200-query subset, and do the ablation rankings hold when both are evaluated by the same judge on the same subset?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.