- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
HITRA studies question answering over long documents using a limited active memory. The agent reads chunks sequentially, updates its current state, and saves historical states that it can retrieve later. Training combines feedback on individual reasoning steps with final-answer rewards. The method also generates multiple reasoning trajectories for each question and learns to rank them using a listwise objective. Experiments on HotpotQA with increasing document counts and controlled diagnostic sets report improvements in memory recovery, credit attribution, and trajectory selection.
Critical strengths
-
The framework connects several relevant failure modes of long-context agents. A compact memory can discard information before its relevance becomes apparent, and a correct final answer provides little guidance about which intermediate decisions were useful. HITRA combines historical-state retrieval, intermediate feedback, and trajectory selection to address these interacting problems within one framework.
-
The diagnostics attempt to explain improvements beyond final-answer accuracy. Separate studies examine recovery after memory corruption, identification of faulty decisions, and selection from a shared candidate pool. This is more informative than a single end-task comparison. Reporting three training seeds for the credit-assignment study also provides some evidence about the stability of those particular results.
Critical weaknesses
-
Two central components substantially overlap with insufficiently discussed prior work. HITRA saves historical memory states for later retrieval and combines step-level rewards with trajectory-level outcomes. ReMemR1 [1] already introduces both mechanisms in the recurrent memory setting represented by MemAgent [2]. Although ReMemR1 appears in Table 1, I could not find a corresponding reference entry or substantive method comparison. The claim of introducing snapshot recall “for the first time” therefore needs revision. Given the additional precedent for probability-based ranking [3], the contribution should be identified more precisely at the level of the new trajectory-ranking design and its integration.
-
The main baseline comparison cannot be fully audited. HITRA uses multiple candidate trajectories and a learned selection mechanism, making inference budget part of the evaluation protocol. Meanwhile, all 16 ReMemR1 entries in Table 1 match its published results [1]. Reusing published results can be appropriate, but the submission does not clearly state their provenance or establish matching examples, metrics, and inference budgets. Without this information, an accuracy gain cannot be cleanly attributed to the proposed learning method.
-
The rewards that define the method are not specified precisely enough. Training uses scores for plan quality, action grounding, structural validity, and exploration-related properties such as new evidence and resolved uncertainty. These scores determine which behavior is reinforced. However, the scorer implementation, prompts, operational scoring rules, and several reward/ranking settings are insufficiently documented. Conceptual descriptions alone are not enough to reconstruct the training objective.
-
The diagnostic labels and interventions need clearer definitions. The recovery study deliberately corrupts intermediate states, while the attribution study evaluates whether the method identifies the responsible decisions and agrees with counterfactual effects. These measurements require a clearly defined corruption process, root-cause labels, and intervention protocol. Their current description is insufficient to assess whether the reported gains reflect general recovery and attribution abilities or properties of the constructed diagnostic sets.
-
The strongest scaling claims exceed the breadth of the evidence. The main accuracy study increases the number of documents around HotpotQA questions, rather than testing several distinct long-context tasks. At 6,400 documents, HITRA improves over ReMemR1 by 0.5 points for 7B and performs slightly worse for 3B. These results support competitive performance on the tested benchmark, but broader improvement claims require uncertainty estimates and evidence beyond this setting. Also, The gains shrink as context grows, which is the opposite of what the memory-recovery motivation predicts. Over ReMemR1, HITRA gains 3.1 to 4.2 points at 50 to 200 documents with 3B, but its margin ranges from −2.8 to +1.6 beyond that and is −0.4 at 6,400 documents. With 7B, the margin falls from about +3.5 at 50 to 200 documents to +0.2 to +0.8 from 800 documents on. This suggests the gains come mostly from short contexts and may not be driven by snapshot recall. The claim of maintaining optimal performance across scales should be scoped accordingly.
-
Several experiments that Appendix C says were run are not reported anywhere in the paper. Appendix C states that a full-graph ListNet run verifies Eq. (4), that varying the microbatch token budget tests packing invariance, and that sweeps over K, chunk size, α and the ranking coefficients separate the gains from sampling, hierarchical credit and listwise training, and it also says paired bootstrap confidence intervals are reported. None of these results appears in any table or figure. The missing sweep over K matters most: HITRA selects one answer from K = 8 trajectories at inference, and only this sweep could show whether the gains come from sampling more candidates.
[1] Look Back to Reason Forward: Revisitable Memory for Long-Context LLM Agents. 2025.
[2] MemAgent: Reshaping Long-Context LLM with Multi-Conv RL-based Memory Agent. 2025.
[3] RRHF: Rank Responses to Align Language Models with Human Feedback. 2023.
Questions for authors
- What improvement comes from adding only HITRA's ranking component to ReMemR1?
5 comments
Response to Reviewer GQAq(2/3)
W4: The diagnostic labels and interventions need clearer definition
Appendices C and F.3 specify that the recovery diagnostic starts from a clean trajectory with a correct final answer. A pre-update state is selected if it contains a reference-answer string and an earlier snapshot retains the same string. The corresponding string in the active state is then replaced with HITRA_CORRUPTED_ENTITY. Valid snapshot labels consist of the identifiers of earlier snapshots that retain the correct claim. Each recall strategy replays the same prefix and continues generation from the same intervention boundary, with information recovery and state correction evaluated separately within eight recurrent updates. Appendix F.4 specifies that the attribution diagnostic injects errors by replacing the plan with UNRELATED_HYPOTHESIS, replacing the action with INVALID_OPERATION, or deleting a closing </plan> tag. An example is accepted when the intervention decreases the corresponding fixed component score, and the ground-truth location label comes directly from the known intervention step and component. Evaluation uses eight candidate trajectories, with the intervention applied to one of them. Responsibility scores are computed by subtracting the intervened trajectory’s component score from the corresponding candidate-group mean. Counterfactual effects are measured through paired repair continuations at each position. The two conditions share the checkpoint, input prefix, budgets, and request seed. Repair gains are calculated as the mean difference in terminal Joint F1, using Answer F1 when supporting-fact annotations are unavailable. Spearman correlation is then computed between the responsibility scores and repair gains.
W5: The strongest scaling claims exceed the breadth of the evidence
The experiment extends the context by adding distractor documents and processing them in 5,000-token chunks, evaluating multi-hop question answering over longer document streams. HITRA combines a bounded compact state, historical snapshot recall, component-wise feedback, and two ranking objectives. These designs support settings with sparse, widely separated evidence and historical information that needs to be recovered after compression. Recall benefits depend on information retention, retrieval accuracy, and effective use of recovered information.The added 2Wiki MultiHopQA experiment evaluates both model scales across different context sizes (Table 1(b)). HITRA outperforms ReMemR1 at both scales in the tested settings with up to 200 documents, while its relative advantage varies as context grows. The 7B model consistently achieves higher accuracy than the 3B model across the evaluated scales.Tables 2–4 examine recall, local credit, and ranking objectives, respectively. Under controlled state corruption using a fixed checkpoint, HITRA achieves better recovery and question-answering quality than No Recall, and outperforms similarity retrieval while using fewer recalled tokens. Table 3 shows that removing local credit or using unfactorized credit weakens final task performance. In Table 4, removing either ranking objective reduces downstream question-answering quality at the fixed candidate budget, with a larger decline when reward ranking is removed.
W6: Several experiments that Appendix C says were run are not reported anywhere in the paper
Appendix E.1 provides a mathematical derivation of the gradient-matching property in Eq. (4) of the main text. Numerical results, such as gradient errors, for the full-graph ListNet verification described in Appendix C have yet to be reported (Eq. (4), Appendices C and E.1). Consistency under microbatch accumulation is ensured by fixed cohort statistics and global normalization denominators, and the appendix provides the corresponding accumulation identity. We set (K=8) as the common candidate budget for training, validation, and inference. The eight synchronized trajectories support within-group feedback standardization, listwise training, and final-answer selection. Increasing the number of candidates also increases trajectory-generation costs. At (K=1), the listwise ranking weights and gradients vanish, so multiple candidates are required for the ranking mechanism to operate.
Response to Reviewer GQAq(3/3)
Q1: Gains from adding only HITRA’s ranking component to ReMemR1
The first step of our HITRA pipeline is to augment recurrent memory with selective recall of earlier state snapshots. The sentence should therefore be revised to “As the first step of HITRA, we augment recurrent memory with selective recall of earlier state snapshots” to accurately convey the order of operations within the method. HITRA emphasizes three specific design elements. First, local feedback is explicitly divided into plan, action, and format channels, each normalized within its cohort before being combined with the trajectory-level overall advantage, incorporating distinct types of local feedback into the training objective. Second, HITRA introduces listwise learning among synchronously generated candidate trajectories, constructing separate ranking targets from overall rewards and breadth signals. The distribution induced by the policy’s own log-likelihood scores is trained to reflect the relative quality of candidate trajectories. Finally, we derive a separable surrogate loss whose gradient matches that of the ranking objective at the behavior-policy parameters, supporting training when candidates are distributed across different microbatches. At inference, the final answer is selected using the same policy’s mean intermediate-turn log-likelihood. Table 1 in the experimental section already includes ReMemR1 as a direct baseline and reports accuracy comparisons across different model scales and numbers of context documents.
Response to Reviewer GQAq(1/3)
Thank you for your valuable comments and careful review. We appreciate your recognition of HITRA’s unified framework for memory recovery, intermediate feedback, and trajectory selection, as well as our use of controlled diagnostics to analyze the mechanisms behind the improvements and our reporting of results across multiple training seeds for the credit-assignment study. Below, we provide point-by-point clarifications regarding related work and methodological novelty, the provenance of baseline results and inference budgets, the reproducibility of rewards and training objectives, diagnostic labels and intervention protocols, the scope of our scaling conclusions, and unreported experimental results and ablations on the number of candidates.
W1: Two central components substantially overlap with insufficiently discussed prior work
The first step of our HITRA pipeline is to augment recurrent memory with selective recall of earlier state snapshots. The sentence should therefore be revised to “As the first step of HITRA, we augment recurrent memory with selective recall of earlier state snapshots” to accurately convey the order of operations within the method. HITRA emphasizes three specific design elements. First, local feedback is explicitly divided into plan, action, and format channels, each normalized within its cohort before being combined with the trajectory-level overall advantage, incorporating distinct types of local feedback into the training objective. Second, HITRA introduces listwise learning among synchronously generated candidate trajectories, constructing separate ranking targets from overall rewards and breadth signals. The distribution induced by the policy’s own log-likelihood scores is trained to reflect the relative quality of candidate trajectories. Finally, we derive a separable surrogate loss whose gradient matches that of the ranking objective at the behavior-policy parameters, supporting training when candidates are distributed across different microbatches. At inference, the final answer is selected using the same policy’s mean intermediate-turn log-likelihood. Table 1 in the experimental section already includes ReMemR1 as a direct baseline and reports Accuracy comparisons across different model scales and numbers of context documents.
W2: The main baseline comparison cannot be fully audited
For each question, HITRA generates eight candidate trajectories, each processing the complete document stream, and selects the answer based on the mean intermediate-turn log-likelihood. Both the candidate count and the selection mechanism are part of the inference budget. HITRA’s Accuracy is computed using exact match (EM) after answer normalization.
W3: The rewards that define the method are not specified precisely enough
Appendices A, B.2, and D.2 provide prompts and examples, field parsing, and the rules for the fixed scoring function
score_turn, respectively. The plan score combines input coverage and gains in answer information, while the action score combines stem matching and evidence coverage. Both scores average three components. Format scoring applies binary checks of field completeness, order, occurrence counts, and nonempty content. Appendix D.3 combines the reciprocal of one plus the snapshot’s cumulative usage count, the proportion of new evidence terms, and the fraction of resolved uncertainty items with weights of 0.25, 0.5, and 0.25, respectively. The latter two components are multiplied by evidence coverage, and the coverage threshold for uncertainty resolution is 0.5. Appendix D.1 sets the plan, action, and format weights to 1, 1, and 0.25, respectively. Each channel is standardized separately within candidate cohorts for the same question and step before aggregation, with a stabilizer of 10−6. The intermediate-turn weights for terminal, local, and breadth rewards are 1, 0.3, and 0.1, while the mixing weights for overall and local advantages are 0.8 and 0.2. The reward-ranking and breadth-ranking coefficients are 0.1 and 0.05, respectively. All ranking temperatures are 1, the KL coefficient is 0.001, and the PPO clipping parameter is 0.2. The complete loss and surrogate-objective derivations are provided in Appendices D.4 and E.