- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes DARS, a credit assignment paradigm for training LLMs on agentic tasks through RL. A prerequisite graph represents task dependency, and then an annotator labels verification, invalidation, and repair events. Constant local monitoring on the graph defines potential, of which the calibrated changes provide step credit alongside the original outcome signal. Experiments cover ALFWorld, WebShop, Search-R1, agentic mathematics, and tool-free reasoning, where the strongest evidence is a matched ALFWorld 1.5B improvement and ablations for attenuation, topology, and credit assignment. Other benefits depend on recipe, scale and training budget.
Critical strengths
- The graph representation is first-principled with how an agentic task should be evaluated.
- The method separates local evidence, broken prerequisites, repair, and re-verification in a structured way. The same-annotation scalar control, attenuation twins, and event-preserving topology comparison test substantive choices in using structured feedback. In particular, serializing WebShop's requirements costs 14.2 points (Table 4 and appendix K). This provides useful knowledge beyond a performance ranking.
- On ALFWorld 1.5B, DARS reaches 96.9% vs. 86.9% for GiGPO under the same 500-step policy budget and harness. The paper also supplies equally extended WebShop ablations, baseline catch-up, and Search component failures (Table 8, 9, 13, 14 and 18). These support a credible and narrower practical contribution.
Critical weaknesses
- Tool-free evaluations combine checkpoints, hosts, and repeats, with unequal mixtures across arms and a different 4B primary-host collection from the pooled summary. Exact schedules, raw scores, decoding seeds, and permutation blocking/exchangeability are not specified (Table 21 and 23). Thus the significance claims are not fully assessable. Most central comparisons also use one training run per arm - decoding repetitions measure those policies, not variation in the training-method effect. The paper acknowledges this, but small cross-domain gains remain uncertain.
- The selected ALFWorld hard-case audit, same-annotator repeatability, and outcome-conditioned consistency checks do not establish population correctness of intermediate V/B/P events. Outside ALFWorld, timing and semantics receive limited independent validation; 29% of usable 4B reasoning annotations omit an answer verdict (Appendix B and J). Since the potential depends on these annotations, this limits the interpretation of its rewards as valid local progress.
Questions for authors
- Can you provide the schedules, raw scores, decoding seeds, and exact permutation design behind Table 21 and 23? Do the principal DARS-scalar and DARS-OmniOPD effects remain when matching existing cells by checkpoint and host?
- What validation supports intermediate event correctness and timing in WebShop, Search-R1, and mathematics? Can inspection/replay of existing annotations quantify errors and omissions and assess sensitivity to correcting or excluding them?
- A frozen-model graph-in-context comparison is interesting additional verification: Just give the untrained model the same graph in context and ask it to solve the task by executing the graph. Can you provide your insights to this idea, and provide some numbers if it makes sense to do so?
6 comments
Author Rebuttal Part 2
W2&Q2: What independent evidence supports the correctness and timing of intermediate events in WebShop, Search-R1, and mathematics, and how sensitive is reward allocation to errors and omissions in those annotations?
Thank you. We distinguish two questions: whether an event is correct, and whether the resulting supervision is useful for training. We have studied annotation quality through environment checks, answer checks, timing interventions and reward replay. These provide evidence about specific aspects of quality; they do not certify every intermediate event.
(1) What can serve as ground truth for V/B/P?
V records that a result has been established, B that an error has been introduced or a condition invalidated, and P that a previous error has been repaired. Validating an event therefore requires both a defined node meaning and evidence at the annotated turn. The original LLM annotation is not ground truth. Environment states and benchmark answers provide external references for some events; open-ended reasoning requires additional semantic judgment. Different valid decompositions and discovery-versus-confirmation timings can also produce different annotations of the same trajectory. Consequently, a disagreement is not automatically an error, but agreement alone does not establish correctness either.
For WebShop, we reconstructed predicate events from goal metadata and page states on 1,280 trajectories per run, covering successes and failures. V precision was 97.8–98.2% when timing was ignored. Exact-turn precision was 22.7–28.6% at one-turn alignment and 29.8–51.4% at zero-turn alignment; within-one-turn precision/recall at one-turn alignment was 57.3–75.9%/29.8–41.7%. These checks support recognition of the covered predicates while identifying timing and recall limitations; they do not validate every B/P event.
For Search-R1, the reference combines the dataset's gold answers, the environment's exact-match scorer and the turn containing the submitted answer. Among 480 stored failed trajectories, 406 contained an incorrect answer submission. New annotations emitted 337 B(answer) events, of which 263 matched: precision = 263/337 = 78.0%, recall = 263/406 = 64.8%. All 263 matches were at the submission turn. These are incorrect-answer-event metrics, not accuracy over all V/B/P events. The annotator was told the trajectory failed; original training annotations were unavailable. The audit annotator version also differs from the corpus annotator. These metrics therefore do not validate historical training labels or intermediate-hop correctness.
For mathematics, our exploratory timing study compared 30 natural distance-two placements from 15 trajectories and two annotator arms with 32 artificially shifted annotations from one arm. GLM flagged 0.0% versus 65.6%, and Astra 6.7% versus 50.0%, as unsupported or incorrectly timed. This is consistent with defensible timing variation and detection of deliberate perturbations. The groups were unpaired, natural placements inherited segment-level judgments, and node IDs lacked fixed semantic definitions. These LLM judgments therefore provide supporting evidence rather than ground truth for intermediate reasoning.
The dependency graph itself is also an inferred representation, not a certified or necessarily unique proof graph. To examine concrete failure modes, we added an Astra-assisted qualitative inspection of eight cached task graphs (four each from WebShop and ALFWorld) and ten stored mathematics re-annotations under the production rubric, deliberately including V-only, B, P and empty-event cases. The cases were selected before the new judgments, and the inputs and reviews are retained. These are separate audit inputs, not identified errors in the exact labels or graph instances of the training comparisons below; the audit describes failure modes rather than estimating population error rates.
Author Rebuttal Part 3
| Component | What the inspection found |
|---|---|
| Graph | Some shopping graphs preserve the requested attributes but serialize independent properties, such as color and capacity, into prerequisites. One inspected graph leaves independent requirements unordered. Requirement coverage and edge correctness need separate checks. |
| V | Correct final answers can be recognized only at a later repetition; a verified answer can also agree with a faulty reference. Intermediate IDs such as step:2 lack exported meanings, preventing definitive node-level validation. |
| B | One truncated geometry solution was assigned B(answer) although it never stated a final answer. An unfinished solution should not automatically receive an incorrect-answer event. |
| P | Two inspected B-to-P sequences coincide with explicit algebra errors and their subsequent corrections. Another repair placement remains ambiguous. Missing node meanings prevent certifying the precise node attribution even when the local error and repair are evident. |
| Omissions | The inspected empty-event cases contain no final answer, so absence of an answer verdict is appropriate; whether useful intermediate work was omitted depends on the unspecified node meanings. |
(2) Can training still be effective, and how much does correcting annotations change credit?
Imperfect supervision can retain useful learning signal. In Li et al., Findings of EMNLP 2025, SFT of Qwen2.5-32B-Instruct on approximately 4,600 reasoning traces with wrong final answers increased average accuracy from 56.7% to 63.1%, compared with 66.3% using correct traces. This supports learning useful reasoning behavior from imperfect examples. In Shao et al., ICML 2026, Qwen2.5-Math-7B gained 21.4 percentage points on MATH-500 with random rewards, versus 29.1 with correct rewards. These results establish that noise and effective training can coexist in specific settings; they do not establish that our annotation errors are random or harmless.
Our evidence for DARS's utility is its held-out evaluation with the annotations actually used. For example, the matched ALFWorld 1.5B evaluation improves success by 4.85 percentage points over GiGPO (game-clustered 95% CI [+1.21, +8.77], one training run per arm). Useful credit also survives changes of representation: four automatically generated WebShop specifications produced credit–environment-progress correlations of 0.357–0.509, versus 0.264 for our specification, on the same 49 failed trajectories within a 120-trajectory pool. Only one difference interval excludes zero. In a separate step-200 comparison, automatically generated graphs achieved 69.21% success versus 69.35% with our rule (paired difference, ours minus generated: 0.14 percentage points, 95% CI [−1.6, +1.8]; one training run per arm). Graded task score still favored our rule by 2.27 points. These results support the utility of the implemented training pipeline and tolerance to the tested specification changes; they do not certify each inferred event or edge.
We quantify credit sensitivity by holding trajectories and the reward computation fixed and changing only annotations. Correcting Search-R1 answer events changed 212/480 = 44.2% of reward vectors. The summed absolute reward change was 42.8% of the original summed absolute reward, and 7.37% of originally nonzero-reward turns changed sign. Replacing WebShop annotations with reconstructed events changed 93.9–95.9% of vectors; randomly deleting 19.7–22.3% of events changed 52.3–55.1%. Random deletion measures omission sensitivity, not targeted removal of adjudicated errors. These are measurable changes in credit allocation, not estimates of lost task accuracy. The causal performance cost of annotation errors remains unmeasured without matched training on original and corrected labels.
Finally, the quoted 29% mathematics omission figure came from a separate audit tape. Answer-verdict absence on the actual training tapes was 59/2,919 = 2.0% at 4B and 317/775 = 40.9% at 1.7B. These are absence rates, not independently established error rates: the rubric requires no answer event when no final answer is stated.
Author Rebuttal Part 1
Dear Reviewer cvNT,
Thank you for your careful reading of our paper and for your constructive feedback. We respond to your comments point by point below.
W1&Q1: How were the tool-free comparisons in Tables 21 and 23 evaluated, how much uncertainty do decoding repeats capture, and do the DARS–scalar and DARS–OmniOPD results hold when checkpoints and hosts are matched?
We release the full evaluation ledger (542 cells, 37 columns), per-arm schedules, per-sample records (Table 23: 82/82 upper, 53/53 lower; Table 21: 62/78), and the exact permutation code. Design: the unit is one evaluation cell, over all relabelings of the two arms' cells, unblocked, with no multiplicity correction — Table 21 by exact enumeration, Table 23 by 200,000 Random(0) draws, which exact enumeration confirms (.0051).
Decoding is uniform and documented for every cell: T=1.0, top_p=1.0, top_k=−1, n=8, max_new_tokens=32768, chat template, thinking on, sglang 0.5.10.post1. The sampler seed was not set, so each repeat is an independent draw rather than a replayable run — which is precisely why we report arm means over cells with permutation tests rather than single cells or maxima.
On matching: the two arms' checkpoint grids coincide at 1.7B step 30; at 4B they do not overlap, so the matched comparison there is by host.
Host-stratifying the 1.7B cells gives +0.95 (p=.040). Every checkpoint-matched pair and every host-matched contrast favours DARS.