- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper introduces new problem named redundant step detection for detecting the steps not contributing to task completion. Redundancy is defined conceptually as a step whose change in task progress is lower than a threshold. It is divided into four types, abnormal (failures cause by external errors), duplicated (re-acquiring information that already obtained), incorrect (violating the task objective, policy or tool constraints), and exploratory (reasonable at that time, but not used). To measure the redundancy, the authors proposed RedundnacyBench, built from existing benchmark in three domains. Human experts and LLMs label the trajectories, and the final benchmark has 1920 trajectories and 3613 step-level annotations. The authors conducted experiments across 6 LLMs, additionally with Jev.
Critical strengths
- The paper proposes a novel concept of research problem. Existing benchmarks only focus on token, latency, or failure attribution, but the authors introduce redundancy, focusing on step-level efficiency. It is clearly important aspect when evaluating the complicated long-horizon agentic tasks.
Critical weaknesses
- Two proposed types are not proper. "Exploratory" is reasonable step at that moment, so the model cannot fix its action in advance. It can be detected after the task is completed. Additionally, "Abnormal" is an external error which agent cannot handle. The concept of redundancy is well motivated, but these two types seem to be inappropriate for redundancy check.
Questions for authors
-
Is there any possibility that the paper including LLM bias? The model generated the label and the model for tested are same family.
-
The authors only report the precision in StepDet, without the recall. Can false positive be computed in trajectories that all of them have at least one redundant step? Furthermore, there could be multiple valid evidence.
2 comments
Rebuttal by Authors!(2/2)
Q2.1: The authors only report the precision in StepDet, without the recall. Can false positive be computed in trajectories that all of them have at least one redundant step?
We only report Precision@K for StepDet task as it measures reliability of detectors by penalizing false positives. Specifically, Precision@K and Recall@K are computed as follows:
\mathrm{Recall@K}=\frac{1}{N}\sum_{i=1}^N\frac{P_i\cap G_i}{|G_i|},$$ where $N$ is the number of trajectories, $P_i$ and $G_i$ denote the set of predicted and ground-truth triplets (step location, category, and supporting evidence) for $i$-th trajectory. A **high Precision@K** indicates a **low false-positive rate** (i.e., necessary steps incorrectly identified as redundant), whereas a high Recall@K indicates a low false-negative rate (i.e., fewer ground-truth redundant steps are missed). For efficiency optimization, false positive predictions are catastrophic than missed redundant steps: removing a necessary step that is incorrectly identified as redundant may lead to task failure. Here we also report Recall@K for StepDet task, as following: | Model | R@1 | R@5 | R@10 | Mean | |---|---:|---:|---:|---:| | GPT-5.4-mini | 2.98 | 5.29 | 5.40 | 4.56 | | Qwen3.8-27B | 14.87 | 20.26 | 20.52 | 18.55 | | GLM-5.3 | 29.93 | 40.54 | 40.66 | 37.04 | | DeepSeek-V4.1-flash | 52.57 | 63.03 | 63.36 | 59.65 | | GPT-5.4 | 44.54 | 60.85 | 61.32 | 55.57 | | Claude-Opus-5 | 55.22 | 76.25 | 76.79 | 69.42 | As shown in the table, most LLMs struggle to achieve high recall, even with a relatively large $K$ (e.g., $K=10$), indicating that they still miss a substantial proportion of ground-truth redundant triplets. **Q2.2: Furthermore, there could be multiple valid evidence.** Yes, a redundant step may be supported by multiple valid evidence. To address this ambiguity, we explicitly define evidence in our annotation guidelines as **the smallest sufficient event or combination of events needed to justify a redundancy label** (see detail in Appendix A). We also provide category-specific explanations to guide annotators in identifying appropriate evidence. In practice, this definition has proven sufficient for annotators to reach a consensus during the annotation process.
Rebuttal by Authors!(1/2)
Thank you for taking the time to review our work. We are pleased that you recognize the novelty of our proposed research problem and acknowledge its importance in evaluating complex, long-horizon agentic tasks. In the following we address your concerns and questions in the order they are asked.
W1: "Exploratory" is reasonable step at that moment, so the model cannot fix its action in advance. It can be detected after the task is completed. Additionally, "Abnormal" is an external error which agent cannot handle. The concept of redundancy is well motivated, but these two types seem to be inappropriate for redundancy check.
We would like to clarify that redundant-step detection is not intended to identify redundancy that model can fix it in advance, but design for post-hoc, offline diagnosis of agent efficiency bottlenecks. This diagnosis covers the entire agent execution pipeline, including LLM, agent harness and external environment. Therefore, we include both Abnormal Steps and Exploratory Steps as redundancy categories: the former represent execution overhead caused by external failures, while the latter capture unnecessary exploratory interactions arising from insufficient information by LLM, neither of which contributes to task completion.
For agent efficiency optimization, we believe that different redundancy categories call for different optimization strategies:
Q1: Is there any possibility that the paper including LLM bias? The model generated the label and the model for tested are same family.
We thank the reviewer for raising this concern. A straightforward way to assess potential LLM bias is to examine whether the performance rankings remain consistent when evaluated on a human-annotated dataset, where the gold labels are provided by human annotators. To investigate this, we compare performance rankings of 6 LLMs on the human-annotated and auto-annotated dataset. Specifically, for each evaluated LLM, we compute the average Prec@K across K={1,5,10} on StepDet task and rank the models accordingly. We observe strong consistency between the two rankings. The only differences are minor rank swaps among the top four models, while Qwen3.8-27B and GPT-5.4-mini retain exactly the same positions. These results suggest that potential biases introduced by LLM-based annotation have a limited impact on the relative performance rankings of the evaluated models. More details about this experiment can be found in our response to Question 3.3 from Reviewer tuQb.