- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
Agentic-DPO samples one-step actions from the current student at expert states. It removes invalid actions and actions that are equivalent to the expert's, and then contrasts the expert action with the most probable remaining action. Training combines length-scaled DPO, an SFT anchor, and schema/recovery augmentation that does not change the policy. Sampled actions are never executed to get labels. The evaluation covers closed-loop agent tasks, static action prediction and several backbones. The main claim is that picking the student's likely mistake matters more than the specific preference loss. This gives large gains over augmented SFT, costs less than online RL, and gives a good starting point for later GRPO.
Critical strengths
S1. The mechanism study rules out real alternative explanations: Tables 4 and 24-28 separate the effects of mined-state selection, negative selection, the preference loss, the pairing unit, refreshing and stabilizers, including own-mining controls. The local propositions explain a restricted first update and are honest about their limits, including the unexplained shortfall at . So the paper tells us more than a simple DPO-versus-SFT comparison.
S2. The practical evaluation is broad and careful: The paper reports validation-based selection, multiple seeds, task-and-seed intervals, compute-limited GRPO comparisons, negative audits, and analyses of long-horizon and off-expert states. It is careful to say "close to GRPO", not "equal to GRPO". The fixed-compute controls and the GRPO initialization study (Table 31) are useful for real training decisions, and the alternate expert source and the blind negative audit test key data assumptions.
Critical weaknesses
W1. The supplied negative miner does not do the selection and filtering described in §3.2:
- The miner does not ask the sampler for log probabilities. When scores are missing, it sets them all to zero, so the "most probable" choice just returns the first eligible candidate. Under the default vLLM V1 settings, scores are indeed missing when log probabilities are not requested. So the runtime version matters directly for the most-probable versus random comparison. I tested this fallback in isolation; I did not reproduce neural sampling with the authors' runtime.
- The filter compares tool names only. So a same-tool action with wrong arguments is thrown away, while a non-empty string that cannot be parsed can survive, because the parser gives it an empty tool name.
- The parser also cannot read the JSON-array format that the supplied renderer produces. I rendered all 3,896 provided expert pairs in the JSON-based formats and parsed them back. The parser recovered the right tool name for none of them. This tests whether the renderer and parser fit together, not how often the model outputs invalid actions.
These issues hit the core selection rule and the equivalence filter, not an optional feature.
W2. The supplied training code changes the stated SFT anchor and augmentation:
- Appendix B says the SFT anchor uses action tokens only. The code also keeps the state and history tokens in the SFT labels and averages their loss. A tokenization and gradient check confirmed that the state prefix does contribute to the anchor.
- Algorithm 1 re-renders each negative with a fresh, independent schema view. The code instead reuses the view chosen during mining. Warm-up still varies the views, so augmentation is not missing completely.
- The training script uses quantized LoRA, a 10x higher learning rate, half as many updates per round, and zero reference scores. The paper uses full bf16 fine-tuning, the smaller learning rate, more updates, and the warmed-up reference.
The zero reference and the same-view training may matter little on their own; the near-linear-loss analysis and Table 27 already suggest this. The real problem is knowing which algorithm was evaluated, especially the selector and the anchor. The supplied material says it is the pipeline used for the reported STB results. It may be an old or simplified release, but this needs to be shown, not assumed.
Questions for authors
Q1. Which exact miner, and which fixed runtime version, produced the main STB results and the selection/filter ablations? Please share a small trace that can be replayed: all candidates, parse status, normalized tool and arguments, scores, and the chosen index. Please include same-tool/wrong-argument and malformed examples. How are the JSON-format outputs parsed?
Q2. Which SFT mask, schema-view sampling, reference and optimization settings were used? If the attached code is a simplified version, please map it to the real one. If the reported runs did use the behavior in the attached code, a focused, otherwise matched comparison of most-probable versus random negatives, with these issues fixed, would be needed to support the claimed mechanism.
4 comments
Follow-up on the updated supplementary material
Thank you for the revised supplement. I have gone through these materials, running the parsers, filters and trace replay, and everything works as described. As far as I can tell, the training scripts also include the action-only SFT losses, the new training views, and the reference models being kept fixed after the warmup period. I consider these points addressed.
One thing I am still unsure about is how the code relates to the reported numbers. From your description, it sounds like you ran additional traces to verify the results. If possible, could you point me to the relevant scripts and hyperparameters for reproducing the STB numbers and the baselines in Table 4? Even a small excerpt of the logs from these runs would be very helpful. I certainly don't expect exhaustive logs or any new runs.
SOURCE_NOTES.txt also seems to describe two options for sampling random negatives. Could you clarify which one was used for the experiments in Table 4?
Thanks again for preparing the additional files. I would simply like to understand a bit better how they connect to the published results.
Connecting the Verification Example to Table 4 Experiments
Thank you for taking the time to run the checks. We are glad they resolved the implementation questions, and we appreciate this helpful follow-up.
Your reading is correct: the traces are additional verification runs. We kept the example small so that the core operations could be checked without running the full experimental suite. The included first-round log belongs to that example; Table 4 reports the larger, ten-seed study described in the paper.
To connect the two directly, src/miner.py implements candidate construction and selection, and src/trainer.py implements the preference objective and SFT anchor. The STB columns in Table 4 use Qwen3.5-2B and measure canonical/perturbed next-action accuracy. Their shared preference-training settings are learning rate 2e-5, batch size 16, 150 updates per round, up to five refresh rounds, K=4, β=0.008, α=0.5, and λ=0.5. Training uses full bf16 fine-tuning with AdamW, zero weight decay, and a constant learning rate after 5% linear warm-up; the reference is frozen after SFT warm-up.
The important connection for the baselines is the matched comparison: every Table 4 row starts from PPA+SFT and follows the same augmentation, update budget, and validation-selection protocol. Blocks (a) and (b) also use the same Agentic-DPO-mined states. Block (a) changes the negative, while block (b) changes the objective, with control-specific coefficients selected on validation. This is how the study separates the benefit of correcting a particular student mistake from the effects of additional training or state selection.
On the random-negative question, Table 4 used uniform sampling over distinct valid, expert-different decisions, after parsing and equivalence filtering. Repeated samples of the same decision do not receive extra weight. The candidate-instance option in SOURCE_NOTES.txt refers only to the auxiliary random-choice field in the additional trace, not to the reported Table 4 baseline. We should have made that distinction clearer in our previous response.
We hope this makes the connection easier to follow. Thank you again for helping us make both the implementation and the experimental comparison clearer.
Follow-up after the clarification
Thank you, this answers my remaining questions. I now understand which training settings you used for Table 4 and how its random-negative baseline differs from the auxiliary choices saved in the verification trace.
I checked the updated package earlier, and its parser, filtering, and trace-replay tests passed. Together with your explanation, this sufficiently addresses my main implementation concerns. Please add these clarifications to the supplementary guide and include the scripts and configurations for the Table 4 variants in the final release.
My original review was positive about the mechanism study and the broader evaluation. Since the implementation concerns were the main reason for my earlier rating, I will raise my rating to 3 (Weak acceptance).
Response to Reviewer Vvge
We thank the reviewer for running the supplementary code and providing precise, testable observations. We used W1/W2 as a checklist for a substantial supplementary update: executable tests, complete candidate-level traces, training records, and a short guide mapping each concern to its implementation and evidence.
The earlier supplementary snapshot was outdated. The updated package provides an executable implementation and an additional StableToolBench/Qwen3.5-2B verification run, allowing the selection and training rules to be inspected directly.
W1/Q1: Scoring, decision filtering, and JSON parsing.
1. Explicit action scoring and replayable selection. The miner parses each candidate into a canonical decision, re-renders it under the mining view, and explicitly computes its summed student action log-probability using a separate vLLM scoring pass. Missing or non-finite eligible scores raise an error; they are never replaced by zero.
The included first-round trace contains 2,400 states and all 9,600 candidates. Replaying the records confirms that all 2,009 retained pairs select the highest-scoring eligible candidate. Each record exposes the raw output, parse status, normalized decision, equivalence flag, score, and chosen index. The verification runtime—vLLM 0.19.0, PyTorch 2.10.0+cu128, and Transformers 5.4.0—is recorded in the trace header.
2. Filtering operates on complete decisions. Equivalence compares the canonical tool and normalized arguments. Same-tool actions with different arguments remain eligible, while unparseable outputs are rejected. In the first-round verification trace, 478 selected negatives use the expert’s tool with different arguments, directly illustrating this distinction. All 23 targeted filtering tests pass, including the reviewer’s same-tool/wrong-argument, malformed-output, and expert-equivalent cases.
3. Renderer–parser compatibility is tested across views. The updated parser explicitly supports the JSON-array format produced by the renderer and maps renamed tools back to canonical names. Round-trip checks pass for all 3,892 usable expert targets under each of the four views, with an additional 15,324/15,324 recovery-context checks.
W2/Q2: Action-only anchor, independent views, and training configuration.
1. Action-only supervision. The SFT warm-up and anchor average loss over action tokens only. The supplied checks include 160 tokenizer-sequence checks, a separate logit-gradient check confirming zero direct SFT gradient at state-predicting positions, and agreement between the optimized action-loss computation and a full-forward test calculation.
2. Independent training views. Mined negatives are stored as canonical decisions. At each training draw, the trainer independently samples a new view and re-renders the state, expert action, and negative together.
3. Explicit reference and optimization settings. The verification implementation uses full-parameter training with bf16 computation, learning rate 2e-5, 150 updates per round, effective batch size 16, and the frozen warm-up student as reference. The resolved configuration and first-round training log are included.
For convenience,
python3 scripts/check_package.pyruns the standard-library checks and complete trace replay;REVIEWER_GUIDE.mdmaps Q1/Q2 to the relevant files.These implementation checks complement the paper’s matched selection studies in Table 4. Together, they make the central mechanism concrete: Agentic-DPO identifies and corrects the student’s strongest sampled competing decision at an expert state, extracting targeted corrective supervision from demonstrations without environment rollouts. We appreciate the reviewer’s careful inspection, which helped us make this mechanism substantially easier to verify.