- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The authors propose a framework for long-horizon agents, designed for online recovery by selecting a strategy from a lightweight DSL and separating environmental premises from behavior. The framework was evaluated on 3 benchmarks and 3 models against the original agent work and 5 recovery baselines, and the results show improved performance over the original agents.
Critical strengths
S1: A well-motivated problem framing, and importantly, unlike the baselines, offering an evaluation on whether an agent's failure is caused by an inaccurate environmental premise S2: A clear and simple-to-follow explanation of the framework S3: In the experimental results, the proposed framework improved task success on the 3 benchmarks and 3 models shown
Critical weaknesses
W1: The results are reported without confidence intervals or paired significance tests. Adding CIs or a paired test to compare the method with the baseline to show significant success would make it stronger
S2: The scope of experiments is narrow, and the ablation, the repeat study, and the extended split each use a single model; running these on all three models, and adding more models or benchmarks, would strengthen the claims of the paper
S3: Running the framework adds additional LLM calls, especially in complex tasks, which require additional costs for the LLMs, but the authors acknowledge this as a limitation; however, baselines are not given an equal compute budget, so a cost-matched comparison would help
Questions for authors
Q1: Can you report CIs or a paired test?
Q2: As mentioned in W2, some of the experiments use only one model each; can you extend them to all three models or add additional models?
We thank the reviewer for the constructive feedback and questions.
W1. Confidence intervals and paired tests
During the rebuttal, we additionally calculated 95% paired bootstrap confidence intervals and conducted two-sided exact McNemar tests to compare ALTER with each baseline. Differences and confidence intervals are in percentage points, with differences computed as ALTER minus the baseline. Statistical significance is assessed at the 0.05 level.
GLM-5
GPT-5.6-Terra
Kimi-K2.5
ALTER’s advantage is statistically significant in 17 of the 18 comparisons.
W2. Experiments across all three models
During the rebuttal, we extended the repeated-run, ablation, and extended-split evaluations across all three models. The additional results are consistent with the findings in the paper.
Repeated runs
Ablation
Extended splits
Across all three studies, ALTER achieves the best performance for all three models. This is consistent with the paper’s conclusion.
W3. Cost-matched comparison
Figure 17 discusses ALTER’s performance with respect to its cost. Across the evaluated models, ALTER achieves better performance with comparable computation budgets.
During the rebuttal, we additionally conducted a cost-matched budget analysis with the strongest baselines. Each baseline has a budget at least as large as ALTER’s. The tables report the budgets alongside pass rates from the available recorded runs.
GLM-5
GPT-5.6-Terra
Kimi-K2.5
Across all three models, ALTER achieves the highest pass rate in this cost-matched budget analysis. These results support the contribution of ALTER’s recovery strategy alongside additional test-time computation.