- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper proposes Informed Recursion, which selects RLM recursion depth based on query structure and selectively verifies intermediate decisions. It aims to reduce errors on exact-count and pairwise aggregation tasks, and evaluates the approach on several long-context and reasoning benchmarks.
Critical strengths
The method is simple and practical, combining adaptive recursion depth with targeted verification. The improvements on OOLONG-Pairs and the depth ablation results demonstrate its potential effectiveness.
Critical weaknesses
-
Limited significance and generalizability: My main concern is that the paper addresses a rather narrow failure mode with relatively straightforward heuristics, without providing much generalizable insight into recursive reasoning. The observation that classification errors can be amplified during aggregation is fairly intuitive, and the proposed routing and verification strategies are not particularly novel. More importantly, the authors themselves acknowledge that the method does not consistently generalize beyond aggregation-sensitive tasks, as also reflected in the mixed results on LongCoT-mini. While the improvements on OOLONG are encouraging, I find it difficult to identify a broader methodological principle or insight that can be applied beyond this specific setting. This significantly limits the contribution of the paper.
-
Insufficient support. The failure analysis lacks direct evidence that classification errors are the main cause of RLM failures. In Table 4, the reported LongCoT-mini overall accuracy (44.90%) conflicts with the five domain accuracies, which imply 46.00% over 50 equally weighted tasks. This needs correction or a clear explanation.
Questions for authors
- How was the 44.90% overall result in Table 4 computed?
- Can the authors provide a more in-depth error analysis to demonstrate that the observed failures are indeed caused by the identified issues, and that the proposed method effectively addresses these specific failure modes?
- Can the authors provide broader experiments to clarify when the proposed mechanisms are effective and how well they generalize beyond aggregation-sensitive tasks?
3 comments
Thank you for the detailed response and additional analysis. I appreciate the authors' efforts to address my concerns. However, my overall assessment of the paper remains unchanged, particularly regarding its limited novelty and generalizability. Therefore, I will maintain my original rating.
Thank you for considering our response and the additional analysis.
We understand that your concerns regarding novelty and generalizability remain. However, we would like to clarify that our contribution is not simply the combination of routing and verification.
Our main contribution is identifying and examining a specific failure mechanism in recursive language agents, where errors in intermediate classifications can lead to much larger errors during aggregation. The additional item-level analysis provides direct evidence of this mechanism. We recovered 1,585 classifications, reconstructed the final pair sets from those classifications, and showed that correcting just two incorrect labels eliminated 202 false-positive pairs. We believe this provides a more concrete understanding of where recursive reasoning fails and where interventions can be useful.
Regarding generalizability, we agree that our method does not improve every reasoning task. However, the evaluation also includes a held-out BrowseComp-Plus setting, where accuracy improved from 82.00% to 89.33%, and CodeQA, where the Full system achieved 66.00% compared with 64.00% for the best fixed-depth baseline. We have reported the negative LongCot-mini results as well, rather than claiming universal improvements.
We recognize that these results may not fully resolve your concerns, but we believe the revised analysis and the results outside OOLONG deserve consideration when assessing the paper's contribution.
Response to Questions on Results, Error Analysis, and Generalization
Thank you for the feedback. We have addressed each question below and have incorporated the relevant corrections and analyses into the revised manuscript.
1)During our final evaluation passes, we identified and corrected a scoring issue affecting one Longcot result. And although the domain-level accuracies were updated, the overall accuracy was mistakenly left at its previous value of 44.90%. The correct value is 46.00% (23/50 correct). We apologise for this oversight. Table 4 has been corrected in the revised manuscript.
2)We revisited the original GPT-5.6 Luna trajectories and performed an item-level analysis of an OOLONG-Pairs query.
Evidence for the failure mechanism: We recovered all 1,585 intermediate classifications from the fixed Depth-2 and Full Informed Recursion executions and compared them against reference labels.
Fixed Depth-2 made 149 classification errors (90.60% accuracy), while Full Informed Recursion made 97 errors (93.88% accuracy). Independently reaggregating the recorded classifications reproduced both final predicted pair sets exactly.
We then performed controlled counterfactual corrections on the Full trajectory. Correcting one incorrect classification eliminated 101 false-positive pairs, increasing F1 from 75.73% to 76.49%. Correcting two eliminated 202 false-positive pairs, increasing F1 to 77.26%. Additionally, 25 of the 97 incorrect classifications individually affected the predicted pair set when corrected in isolation.
These results directly demonstrate how intermediate classification errors can propagate through membership conditions into larger aggregation errors.
Evidence for the proposed approach: On the same query, Full IR achieved 75.73% pair F1, compared with 67.44% for fixed Depth-2, along with the reduction from 149 to 97 classification errors.
This provides task-level evidence that the combined configuration can improve both intermediate classification accuracy and final outputs. The analysis establishes the propagation mechanism in this case, rather than claiming that it is the sole cause of errors across the benchmark. We have Added this analysis to Section 3.3.
3)We have noted that the effectiveness of Informed Recursion must be distinguished from its generalizability.
Our evaluation includes long-context aggregation, repository comprehension, multi-domain reasoning, and held-out multi-document retrieval. The results show a task-dependent pattern:
1)OOLONG-Pairs: 78.31% F1 for Full Informed Recursion versus 70.53% for fixed Depth-2. 2) OOLONG: 66.50% acc vs 65.97% for fixed Depth-2. 3) CodeQA: 66.00% acc vs 64.00% for Depth-1 and 62.00% for Depth-2. 4) Longcot-mini: 46.00% for Full IR vs 48.00% for Depth-1. 5) BrowseComp-Plus: 89.33% acc vs 82.00% for the evaluated RLM baseline.
These findings indicate that the strongest observed improvement occurs on OOLONG-Pairs. Results on other reasoning tasks are more variable, as shown by LongCoT-mini. Our approach does not improve performance across all reasoning tasks.
This distinction has been clarified in Section 6.4 and the conclusion now emphasizes the aggregation-sensitive setting rather than claiming a general improvement to recursive reasoning.