acceptodds
Under review as a conference paper at ICLR 2027

HITRA: HIERARCHICAL TRAJECTORY RERANKING FOR LONG-CONTEXT AGENT

Abstract

Reliable multi-hop question answering over thousands of documents requires agents to connect sparse evidence across long observation sequences under a limited active-context budget. Existing approaches largely follow a modular trajectory-processing paradigm, addressing memory, decision evaluation, and candidate selection. Although effective within their respective roles, they leave a coupled gap: compressed states can lose recoverable hypotheses, intermediate feedback can obscure distinct decision failures, and selection lacks a jointly trained ordering over partial trajectories. To address these issues, this paper proposes Hierarchical Trajectory Reranking for Long-Context Agent (HITRA), a framework designed to jointly address the interrelated challenges of memory compression, intermediate decision evaluation, and partial-trajectory selection. Especially, selective snapshot recall equips compact recurrent rollouts with recoverable prior hypotheses, allowing later evidence to correct state errors within a bounded active context. Hierarchical credit separately normalizes planning, action, and structural-validity feedback before combining local quality with terminal utility and exploration, distinguishing decision failures while aligning updates with task success. Listwise learning explicitly orders concurrent partial trajectories, with a separable surrogate that supports microbatch training and exactly matches the full listwise gradient at the behavior policy. Together, these methods enable evidence recovery, finer-grained attribution, and listwise ranking of concurrent partial trajectories. Extensive experiments demonstrate that across context sizes ranging from 50 to 6,400 documents, HITRA maintains (near-) optimal performance across all evaluated context scales. Controlled diagnostics further demonstrate improvements in memory recovery, root-cause localization, and trajectory selection: Compared to all baselines, the memory recovery rate increases by at least 8.7 percentage points (see Table 2 Recovery Rate), the root-cause localization rate increases by at least 9.7 percentage points (see Table 3 Root@1), and the optimal candidate selection capability improves by at least 18.0 percentage points (see Table 4 Hit@1).

Primary area:
reinforcement learning
Keywords:
Long-Context Agents, Multi-Hop Question Answering, Trajectory Reranking, Listwise Learning, Hierarchical Credit Assignment

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_GQAqOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

HITRA studies question answering over long documents using a limited active memory. The agent reads chunks sequentially, updates its current state, and saves historical states that it can retrieve later. Training combines feedback on individual reasoning steps with final-answer rewards. The method also generates multiple reasoning trajectories for each question and learns to rank them using a listwise objective. Experiments on HotpotQA with increasing document counts and controlled diagnostic sets report improvements in memory recovery, credit attribution, and trajectory selection.

Critical strengths
  1. The framework connects several relevant failure modes of long-context agents. A compact memory can discard information before its relevance becomes apparent, and a correct final answer provides little guidance about which intermediate decisions were useful. HITRA combines historical-state retrieval, intermediate feedback, and trajectory selection to address these interacting problems within one framework.

  2. The diagnostics attempt to explain improvements beyond final-answer accuracy. Separate studies examine recovery after memory corruption, identification of faulty decisions, and selection from a shared candidate pool. This is more informative than a single end-task comparison. Reporting three training seeds for the credit-assignment study also provides some evidence about the stability of those particular results.

Critical weaknesses
  1. Two central components substantially overlap with insufficiently discussed prior work. HITRA saves historical memory states for later retrieval and combines step-level rewards with trajectory-level outcomes. ReMemR1 [1] already introduces both mechanisms in the recurrent memory setting represented by MemAgent [2]. Although ReMemR1 appears in Table 1, I could not find a corresponding reference entry or substantive method comparison. The claim of introducing snapshot recall “for the first time” therefore needs revision. Given the additional precedent for probability-based ranking [3], the contribution should be identified more precisely at the level of the new trajectory-ranking design and its integration.

  2. The main baseline comparison cannot be fully audited. HITRA uses multiple candidate trajectories and a learned selection mechanism, making inference budget part of the evaluation protocol. Meanwhile, all 16 ReMemR1 entries in Table 1 match its published results [1]. Reusing published results can be appropriate, but the submission does not clearly state their provenance or establish matching examples, metrics, and inference budgets. Without this information, an accuracy gain cannot be cleanly attributed to the proposed learning method.

  3. The rewards that define the method are not specified precisely enough. Training uses scores for plan quality, action grounding, structural validity, and exploration-related properties such as new evidence and resolved uncertainty. These scores determine which behavior is reinforced. However, the scorer implementation, prompts, operational scoring rules, and several reward/ranking settings are insufficiently documented. Conceptual descriptions alone are not enough to reconstruct the training objective.

  4. The diagnostic labels and interventions need clearer definitions. The recovery study deliberately corrupts intermediate states, while the attribution study evaluates whether the method identifies the responsible decisions and agrees with counterfactual effects. These measurements require a clearly defined corruption process, root-cause labels, and intervention protocol. Their current description is insufficient to assess whether the reported gains reflect general recovery and attribution abilities or properties of the constructed diagnostic sets.

  5. The strongest scaling claims exceed the breadth of the evidence. The main accuracy study increases the number of documents around HotpotQA questions, rather than testing several distinct long-context tasks. At 6,400 documents, HITRA improves over ReMemR1 by 0.5 points for 7B and performs slightly worse for 3B. These results support competitive performance on the tested benchmark, but broader improvement claims require uncertainty estimates and evidence beyond this setting. Also, The gains shrink as context grows, which is the opposite of what the memory-recovery motivation predicts. Over ReMemR1, HITRA gains 3.1 to 4.2 points at 50 to 200 documents with 3B, but its margin ranges from −2.8 to +1.6 beyond that and is −0.4 at 6,400 documents. With 7B, the margin falls from about +3.5 at 50 to 200 documents to +0.2 to +0.8 from 800 documents on. This suggests the gains come mostly from short contexts and may not be driven by snapshot recall. The claim of maintaining optimal performance across scales should be scoped accordingly.

  6. Several experiments that Appendix C says were run are not reported anywhere in the paper. Appendix C states that a full-graph ListNet run verifies Eq. (4), that varying the microbatch token budget tests packing invariance, and that sweeps over K, chunk size, α and the ranking coefficients separate the gains from sampling, hierarchical credit and listwise training, and it also says paired bootstrap confidence intervals are reported. None of these results appears in any table or figure. The missing sweep over K matters most: HITRA selects one answer from K = 8 trajectories at inference, and only this sweep could show whether the gains come from sampling more candidates.

[1] Look Back to Reason Forward: Revisitable Memory for Long-Context LLM Agents. 2025.

[2] MemAgent: Reshaping Long-Context LLM with Multi-Conv RL-based Memory Agent. 2025.

[3] RRHF: Rank Responses to Align Language Models with Human Feedback. 2023.

Questions for authors
  1. What improvement comes from adding only HITRA's ranking component to ReMemR1?
5 comments
AuthorsOpenReview ↗

Response to Reviewer GQAq(1/3)

Thank you for your valuable comments and careful review. We appreciate your recognition of HITRA’s unified framework for memory recovery, intermediate feedback, and trajectory selection, as well as our use of controlled diagnostics to analyze the mechanisms behind the improvements and our reporting of results across multiple training seeds for the credit-assignment study. Below, we provide point-by-point clarifications regarding related work and methodological novelty, the provenance of baseline results and inference budgets, the reproducibility of rewards and training objectives, diagnostic labels and intervention protocols, the scope of our scaling conclusions, and unreported experimental results and ablations on the number of candidates.

W1: Two central components substantially overlap with insufficiently discussed prior work

The first step of our HITRA pipeline is to augment recurrent memory with selective recall of earlier state snapshots. The sentence should therefore be revised to “As the first step of HITRA, we augment recurrent memory with selective recall of earlier state snapshots” to accurately convey the order of operations within the method. HITRA emphasizes three specific design elements. First, local feedback is explicitly divided into plan, action, and format channels, each normalized within its cohort before being combined with the trajectory-level overall advantage, incorporating distinct types of local feedback into the training objective. Second, HITRA introduces listwise learning among synchronously generated candidate trajectories, constructing separate ranking targets from overall rewards and breadth signals. The distribution induced by the policy’s own log-likelihood scores is trained to reflect the relative quality of candidate trajectories. Finally, we derive a separable surrogate loss whose gradient matches that of the ranking objective at the behavior-policy parameters, supporting training when candidates are distributed across different microbatches. At inference, the final answer is selected using the same policy’s mean intermediate-turn log-likelihood. Table 1 in the experimental section already includes ReMemR1 as a direct baseline and reports Accuracy comparisons across different model scales and numbers of context documents.

W2: The main baseline comparison cannot be fully audited

For each question, HITRA generates eight candidate trajectories, each processing the complete document stream, and selects the answer based on the mean intermediate-turn log-likelihood. Both the candidate count and the selection mechanism are part of the inference budget. HITRA’s Accuracy is computed using exact match (EM) after answer normalization.

W3: The rewards that define the method are not specified precisely enough

Appendices A, B.2, and D.2 provide prompts and examples, field parsing, and the rules for the fixed scoring function score_turn, respectively. The plan score combines input coverage and gains in answer information, while the action score combines stem matching and evidence coverage. Both scores average three components. Format scoring applies binary checks of field completeness, order, occurrence counts, and nonempty content. Appendix D.3 combines the reciprocal of one plus the snapshot’s cumulative usage count, the proportion of new evidence terms, and the fraction of resolved uncertainty items with weights of 0.25, 0.5, and 0.25, respectively. The latter two components are multiplied by evidence coverage, and the coverage threshold for uncertainty resolution is 0.5. Appendix D.1 sets the plan, action, and format weights to 1, 1, and 0.25, respectively. Each channel is standardized separately within candidate cohorts for the same question and step before aggregation, with a stabilizer of 10−610^{-6}. The intermediate-turn weights for terminal, local, and breadth rewards are 1, 0.3, and 0.1, while the mixing weights for overall and local advantages are 0.8 and 0.2. The reward-ranking and breadth-ranking coefficients are 0.1 and 0.05, respectively. All ranking temperatures are 1, the KL coefficient is 0.001, and the PPO clipping parameter is 0.2. The complete loss and surrogate-objective derivations are provided in Appendices D.4 and E.

AuthorsOpenReview ↗

Response to Reviewer GQAq(2/3)

W4: The diagnostic labels and interventions need clearer definition

Appendices C and F.3 specify that the recovery diagnostic starts from a clean trajectory with a correct final answer. A pre-update state is selected if it contains a reference-answer string and an earlier snapshot retains the same string. The corresponding string in the active state is then replaced with HITRA_CORRUPTED_ENTITY. Valid snapshot labels consist of the identifiers of earlier snapshots that retain the correct claim. Each recall strategy replays the same prefix and continues generation from the same intervention boundary, with information recovery and state correction evaluated separately within eight recurrent updates. Appendix F.4 specifies that the attribution diagnostic injects errors by replacing the plan with UNRELATED_HYPOTHESIS, replacing the action with INVALID_OPERATION, or deleting a closing </plan> tag. An example is accepted when the intervention decreases the corresponding fixed component score, and the ground-truth location label comes directly from the known intervention step and component. Evaluation uses eight candidate trajectories, with the intervention applied to one of them. Responsibility scores are computed by subtracting the intervened trajectory’s component score from the corresponding candidate-group mean. Counterfactual effects are measured through paired repair continuations at each position. The two conditions share the checkpoint, input prefix, budgets, and request seed. Repair gains are calculated as the mean difference in terminal Joint F1, using Answer F1 when supporting-fact annotations are unavailable. Spearman correlation is then computed between the responsibility scores and repair gains.

W5: The strongest scaling claims exceed the breadth of the evidence

The experiment extends the context by adding distractor documents and processing them in 5,000-token chunks, evaluating multi-hop question answering over longer document streams. HITRA combines a bounded compact state, historical snapshot recall, component-wise feedback, and two ranking objectives. These designs support settings with sparse, widely separated evidence and historical information that needs to be recovered after compression. Recall benefits depend on information retention, retrieval accuracy, and effective use of recovered information.The added 2Wiki MultiHopQA experiment evaluates both model scales across different context sizes (Table 1(b)). HITRA outperforms ReMemR1 at both scales in the tested settings with up to 200 documents, while its relative advantage varies as context grows. The 7B model consistently achieves higher accuracy than the 3B model across the evaluated scales.Tables 2–4 examine recall, local credit, and ranking objectives, respectively. Under controlled state corruption using a fixed checkpoint, HITRA achieves better recovery and question-answering quality than No Recall, and outperforms similarity retrieval while using fewer recalled tokens. Table 3 shows that removing local credit or using unfactorized credit weakens final task performance. In Table 4, removing either ranking objective reduces downstream question-answering quality at the fixed candidate budget, with a larger decline when reward ranking is removed.

W6: Several experiments that Appendix C says were run are not reported anywhere in the paper

Appendix E.1 provides a mathematical derivation of the gradient-matching property in Eq. (4) of the main text. Numerical results, such as gradient errors, for the full-graph ListNet verification described in Appendix C have yet to be reported (Eq. (4), Appendices C and E.1). Consistency under microbatch accumulation is ensured by fixed cohort statistics and global normalization denominators, and the appendix provides the corresponding accumulation identity. We set (K=8) as the common candidate budget for training, validation, and inference. The eight synchronized trajectories support within-group feedback standardization, listwise training, and final-answer selection. Increasing the number of candidates also increases trajectory-generation costs. At (K=1), the listwise ranking weights and gradients vanish, so multiple candidates are required for the ranking mechanism to operate.

AuthorsOpenReview ↗

Response to Reviewer GQAq(3/3)

Q1: Gains from adding only HITRA’s ranking component to ReMemR1

The first step of our HITRA pipeline is to augment recurrent memory with selective recall of earlier state snapshots. The sentence should therefore be revised to “As the first step of HITRA, we augment recurrent memory with selective recall of earlier state snapshots” to accurately convey the order of operations within the method. HITRA emphasizes three specific design elements. First, local feedback is explicitly divided into plan, action, and format channels, each normalized within its cohort before being combined with the trajectory-level overall advantage, incorporating distinct types of local feedback into the training objective. Second, HITRA introduces listwise learning among synchronously generated candidate trajectories, constructing separate ranking targets from overall rewards and breadth signals. The distribution induced by the policy’s own log-likelihood scores is trained to reflect the relative quality of candidate trajectories. Finally, we derive a separable surrogate loss whose gradient matches that of the ranking objective at the behavior-policy parameters, supporting training when candidates are distributed across different microbatches. At inference, the final answer is selected using the same policy’s mean intermediate-turn log-likelihood. Table 1 in the experimental section already includes ReMemR1 as a direct baseline and reports accuracy comparisons across different model scales and numbers of context documents.

Reviewer_pmbgOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes HITRA for long-context multi-hop QA. HISTRA is built on recurrent memory agents such as MemAgent, which read the context chunk by chunk and keep a short running summary. The method makes three changes to this setup: (1) Every past summary is saved, and the model can retrieve an earlier one later, so information overwritten by mistake can be recovered. (2) During RL training, each step gets separate rewards for the quality of its plan, the grounding of its action, and its output format. These rewards are normalized GRPO-style within the group of rollouts for the same question and step, and then combined with the final-answer reward and an exploration bonus. (3) An additional ranking loss trains the model’s own likelihood scores to prefer better rollouts. At inference time, the model samples eight rollouts and returns the answer from the one it scores highest. Experiments on HotpotQA with Qwen2.5-3B and 7B show improvements over MemAgent and ReMemR1, along with controlled analyses of memory recovery, error localization, and rollout selection.

Critical strengths

The paper targets a practical weakness of recurrent memory agents: once information is overwritten, it cannot be recovered. The method is described formally. Regarding answer selection, utilizing the model’s own likelihood avoids the need for a separate reward model or reranker.

Critical weaknesses
  1. ReMemR1 is not cited or discussed in the paper, although it is one of the baselines. As far as I know, it already lets the agent retrieve earlier memories and uses multi-level rewards during RL training. Without method discussion or comparison, it is hard to tell what differs HITRA, and the claim that this is the first work to add recall to recurrent memory does not seem justified.

  2. Some experiemental evidences are not very supportive. HITRA selects one answer out of eight rollouts at inference, but it is unclear whether the baselines received the same budget. The diagnostic studies in Tables 2 to 4 do not explain how the corrupted states or the ground-truth labels were created, leave several metrics undefined, and compare mainly against internal variants. Some key details are also missing, including which model scores the plans and actions, how the exploration bonus is computed, and most of the loss weights.

Questions for authors
  1. How does HITRA differ from ReMemR1 in its recall mechanism and reward design? Can you report an ablation that removes only snapshot recall, so its contribution can be separated from the rest of the method?

  2. How were the diagnostic datasets built, how are the metrics in Tables 2 to 4 defined, and which model provides the plan and action scores? Did the baselines receive comparable step-level supervision?

5 comments
AuthorsOpenReview ↗

Response to Reviewer pmbg(1/3)

Thank you for your valuable comments and careful review. We appreciate your recognition of our focus on the practical difficulty of recovering overwritten information in recurrent memory, the formal presentation of our method, and the use of the model’s own likelihood scores for answer selection. Below, we provide point-by-point clarifications regarding the relationship with ReMemR1 and our novelty claims, the comparability of inference budgets and step-level supervision, diagnostic datasets and metric definitions, the scorer, exploration bonus, loss weights, and the snapshot recall ablation.

W1: Insufficient comparison with ReMemR1 and unclear methodological novelty

The first step of the HITRA pipeline is to augment recurrent memory with selective recall of earlier state snapshots. The sentence should therefore read “As the first step of HITRA, we augment recurrent memory with selective recall of earlier state snapshots” to accurately convey the order of operations within the method. HITRA emphasizes three specific design elements. First, local feedback is explicitly divided into plan, action, and format channels, each normalized within its cohort before being combined with the trajectory-level overall advantage, incorporating distinct types of local feedback into the training objective. Second, HITRA introduces listwise learning among synchronously generated candidate trajectories, constructing separate ranking targets from overall rewards and breadth signals. The distribution induced by the policy’s own log-likelihood scores is trained to reflect the relative quality of candidate trajectories. Finally, we derive a separable surrogate loss whose gradient matches that of the ranking objective at the behavior-policy parameters, supporting training when candidates are distributed across different microbatches. At inference, the final answer is selected using the same policy’s mean intermediate-turn log-likelihood. Table 1 in the experimental section already includes ReMemR1 as a direct baseline and reports Accuracy comparisons across different model scales and numbers of context documents.

​

ScaleMethod501002004008001,6003,2006,400
3BQwen2.5 (Yang et al., 2024)39.839.139.0—————
3BMemAgent (Yu et al., 2026)41.445.340.239.436.328.926.725.9
3BReMemR1 (Shi et al., 2026)53.550.442.541.737.036.235.437.8
3BHITRA (ours)57.153.945.342.139.437.535.934.4
7BQwen2.5 (Yang et al., 2024)53.949.261.1—————
7BR1-Distill (DeepSeek-AI et al., 2025)36.729.725.80.00.82.32.30.8
7BQwen2.5-1M (Yang et al., 2025)62.559.457.847.746.145.325.80.0
7BMemAgent (Yu et al., 2026)61.757.850.847.650.744.546.944.7
7BReMemR1 (Shi et al., 2026)63.963.155.654.554.745.448.950.3
7BHITRA (ours)66.465.457.055.954.747.647.246.1
AuthorsOpenReview ↗

Response to Reviewer pmbg(2/3)

W2: Unclear evaluation fairness and insufficient experimental and training details

By default, HITRA generates $K=8$ trajectories per question, each processing the complete document stream, with a generation limit of 2,048 tokens for every intermediate turn and the final-answer turn. Table 2 uses a single trajectory per intervened record, Table 3 uses eight trajectories with an intervention applied to only one, and Table 4 compares selection rules using saved records of eight candidates. The procedure described in Appendices C and F.3 selects records from clean trajectories with correct final answers, requiring both the active state and an earlier snapshot to contain the reference answer. The corresponding content in the active state is replaced with `HITRA_CORRUPTED_ENTITY`, and all recall strategies continue generation from the same boundary. Valid-snapshot labels are derived from earlier snapshots that retain the correct claim. The attribution diagnostic injects errors by replacing plans or actions, or deleting a plan closing tag, with a decrease in the corresponding score as an inclusion criterion. Ground-truth labels are determined by the injection step and component. Appendices F.3–F.5 specify the evaluation targets, formulas, units, aggregation procedures, and boundary rules for memory-recovery, credit-attribution, and trajectory-selection metrics, respectively. In particular, $\rho_{\mathrm{CF}}$ uses per-example Spearman correlation, while Leakage measures the proportion of absolute responsibility mass assigned outside the injected location. The fixed-rule function `score_turn` computes plan and action scores using term coverage, progress in answer coverage, and action-stem matching, averaging three components for each score. Format scoring checks field completeness, order, occurrence counts, and nonempty content, with detailed rules provided in Appendix D.2. At inference, answer selection uses the policy’s own intermediate-turn log-likelihood. The weights for snapshot use, novelty, and uncertainty resolution are $0.25$, $0.5$, and $0.25$, respectively. The snapshot-use term is the reciprocal of one plus the cumulative recall count, including the current recall. Novelty measures the proportion of evidence terms that are new relative to the previous state. Uncertainty resolution checks whether previous items have disappeared and whether their coverage in the evidence reaches the threshold. Both the novelty and uncertainty-resolution terms are multiplied by the evidence’s coverage in the current observation.

Q1: Comparison with ReMemR1 and snapshot recall ablation

ReMemR1 retrieves historical memories based on query-term coverage. HITRA maintains a private snapshot store for each trajectory, preserving pre-update states and provenance information. It generates queries through <expand> and selects snapshots using cosine similarity between term-frequency vectors, supplying token-limited recalled content to the next turn. ReMemR1 combines trajectory-level correctness rewards with step-level memory, recall, and format rewards. HITRA standardizes plan, action, and format feedback separately, combines overall and local advantages, and constructs two listwise ranking objectives from overall rewards and breadth signals. The No Recall control in Table 2 uses the same checkpoint and controlled records, sets the recalled payload to empty, and continues document processing and state updates. The experiment uses Qwen2.5-3B-Instruct on 1,000 state-corruption examples, with a single trajectory per example.

AuthorsOpenReview ↗

Response to Reviewer pmbg(3/3)

Q2: Diagnostic dataset construction, metric definitions, scoring methods, and supervision comparability

For the diagnostic in Table 2, records are selected from clean trajectories with correct final answers, requiring both the pre-update state and an earlier snapshot to contain the reference answer. The corresponding string in the active state is replaced with HITRA_CORRUPTED_ENTITY. All recall strategies use the same records, with a single trajectory for each of the 1,000 examples and a recovery window of eight updates (Appendices C and F.3). In Table 3, a plan, action, or format error is injected into one of eight candidate trajectories. Examples are accepted only when the corresponding score decreases, with the injection step and component providing the ground-truth labels. Paired repair continuations measure changes in terminal utility (Appendix F.4). Table 4 compares selection rules on the eight saved candidates per question, using Joint F1 as gold utility and Answer F1 when supporting-fact annotations are unavailable (Appendix F.5). The diagnostic metrics in Tables 2–4 are detailed in Appendices F.3–F.5, respectively, covering definitions, formulas, aggregation procedures, and boundary handling. Answer EM, Answer F1, Support F1, and Joint F1 are defined in Appendix F.2. Plan and action scores are computed by the fixed-rule function score_turn, with semantic_scorer=null by default. Scoring uses text coverage after lowercasing, term extraction, and deduplication, together with progress in answer coverage and action-stem matching. Each score averages three components and lies in [0,1][0,1], with detailed rules provided in Appendix D.2.

Reviewer_HjrmOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes a long-context agent for multi-hop QA reasoning that combines 1) selective snapshot recall to keep track of previous hypotheses to address the limit of LLM contexts size, 2) hierarchical credit assignment to balance the exploration vs task outcome budgets and 3) listwise trajectory learning for ranking candidate trajectories.

The idea is clear, ambitious, and somewhat comprehensive given the current state of the art. The critical aspects of the paper concern the technical presentation and its unclear details that hinder the proposed contribution.

Critical strengths
  1. The idea put forward by HITRA is timely and ambitious. It aims to solve a complex problem in comprehensive manner, considering the conflicting constraints, that, often are difficult to satisfy.

  2. Results on the single dataset and family of LLMs are positive. Experiments provide a positive general outlook of the proposed idea.

Critical weaknesses
  1. The paper is clear in its goals and framing. Introduction and related work are pleasant to read, but the technical presentation is quite dense with an excess of notation and variables that are not clearly defined creating many ambiguities.

  2. Generalization is not sufficiently convincing. HITRA was validated in only one benchmark (HotPotQA) and only one LLM family (Qwen). There many other models and benchmarks where HITRA success could have been demonstrated (e.g. Bamboogle, MuSiQue, PopQA, TriviaQA). While current experiments are positive, they are not able to provide a comprehensive and in depth picture of HITRA's capabilities.

  3. The complexity of the many different components is significant. However, the contribution and role of HITRA's individual components to the final output is not clear. More comprehensive ablations would be required.

Questions for authors
  • The definition of what compact states z_t entails is not completely clear. The same for snapshots m_t, could you please clarity, in particular its initialization? Variables s are not named. Whole paragraph of line 144 should be clarified.

  • What is a feedback channel?

  • How is reward vs breath ("terminal success" vs "exploration") balanced during training?

  • How are planing, action, format scores obtained? (line 244)

  • How does HITRA selects the eight chunks from the 50-6400 documents?

  • How are the context 50-6400 documents selected?

  • What is the computational overhead of the proposed framework? An analysis similar to MemR1 in terms of would help readers have a better sense of HITRA's properties.

5 comments
AuthorsOpenReview ↗

Response to Reviewer Hjrm(1/4)

Thank you for your valuable comments and careful review. We appreciate your recognition of the timeliness of HITRA’s research topic, its comprehensive approach to a complex problem, the clarity of its objectives and framework, the readability of the Introduction and Related Work sections, and the positive experimental results. Below, we provide point-by-point clarifications regarding your concerns about the technical presentation and symbol definitions, generalization evaluation, component ablations, state initialization, feedback and reward design, context construction, and computational overhead.

W1: Insufficient explanations of some symbols

We have revised the manuscript to address the dense technical presentation and insufficient explanations of some symbols. First, we clarify notation and state transitions. Section 3 defines the indices for tasks, steps, and trajectory replicas, along with the compact state, snapshot store, and recalled payload. Section 4.1 and Appendix A explain the correspondence between states and output fields, specifying that snapshots preserve pre-update states and that recall results from the current step enter the next input. Second, we clarify rewards and advantages. Section 4.2 and Appendix D provide the scoring rules, value ranges, and within-cohort normalization procedures for planning, actions, and structural validity. They explain the roles of local rewards, local advantages, and overall utility, and specify the feedback and masking rules for intermediate steps and final responses. Finally, we clarify the connection between scoring and listwise learning. Section 4.3 explains that each step is scored by the average log probability of its response tokens, while inference-time trajectory scoring averages these scores over intermediate steps. Appendix E explains ranking targets, candidate weights, and microbatch computation, and clarifies that the surrogate and full listwise objectives have identical gradients at the behavior-policy parameters. We have also added explanations for the symbols in Figure 2.

W2: limitations of the generalization evaluation

We have added an experiment on 2Wiki MultiHopQA using 3B and 7B models across 50–6,400 context documents, as reported in Table 1(b). HITRA outperforms ReMemR1 at both model scales in the evaluated settings with up to 200 documents. Across all tested context scales, the 7B variant consistently achieves higher accuracy than the 3B variant.

​

ScaleMethod501002004008001,6003,2006,400
3BQwen2.5 (Yang et al., 2024)39.839.139.0—————
3BMemAgent (Yu et al., 2026)41.445.340.239.436.328.926.725.9
3BReMemR1 (Shi et al., 2026)53.550.442.541.737.036.235.437.8
3BHITRA (ours)57.153.945.342.139.437.535.934.4
7BQwen2.5 (Yang et al., 2024)53.949.261.1—————
7BR1-Distill (DeepSeek-AI et al., 2025)36.729.725.80.00.82.32.30.8
7BQwen2.5-1M (Yang et al., 2025)62.559.457.847.746.145.325.80.0
7BMemAgent (Yu et al., 2026)61.757.850.847.650.744.546.944.7
7BReMemR1 (Shi et al., 2026)63.963.155.654.554.745.448.950.3
7BHITRA (ours)66.465.457.055.954.747.647.246.1
AuthorsOpenReview ↗

Response to Reviewer Hjrm(2/4)

W3: more comprehensive analysis of each component’s contribution

Tables 2–4 examine memory recall, local credit assignment, and the two ranking objectives, respectively, reporting both mechanism metrics and final question-answering quality. Table 2 compares recall strategies under controlled state corruption using a fixed checkpoint. HITRA achieves better information recovery and answer quality than No Recall, and outperforms similarity retrieval while using fewer recalled tokens. Table 3 shows that removing local credit weakens final task performance. Compared with the unfactorized variant, the full method identifies injected errors more accurately, reduces attribution leakage, and achieves higher Joint F1. Table 4 evaluates the ranking objectives with eight candidate trajectories. Removing either objective weakens candidate selection and downstream question-answering quality, with a larger performance decrease when reward ranking is removed.

Q1: Unclear definitions and initialization of compact states, snapshots, and policy inputs

In Section 3, we define the compact state ztkz_t^k retained by the kk-th independent trajectory of question qq after processing document chunk ctc_t. In Appendix A, we specify that it consists of trajectory and uncertainty and is initialized to No trajectory state yet.. Appendix B.2 describes how these fields are serialized, while Appendix C specifies their combined budget of 2,048 tokens.

In Section 4.1 and Appendix B.1, we define the snapshot record νtk\nu_t^k, which stores the pre-update state zt−1kz_{t-1}^k, together with the snapshot and trajectory identifiers, creation step, predecessor identifier, and provenance information. These records form the private snapshot store Mtk\mathcal{M}_t^k. In Section 3, we distinguish these records from mtkm_t^k, the token-limited recalled payload supplied in the current input. We specify the initial conditions M1k=∅\mathcal{M}_1^k=\varnothing and m1k=∅m_1^k=\varnothing in Section 3 and Appendix A, where empty recall is rendered as No snapshot was recalled..

We clarify in Section 4.1 and Appendix B.1 that retrieval searches the existing snapshot store before the current pre-update state is inserted, and that the retrieved content enters the next turn’s input. In Section 3, we define the policy input as stk=(q,ct,zt−1k,mtk)s_t^k=(q,c_t,z_{t-1}^k,m_t^k) and the final-answer input as sNq+1k=(q,zNqk,mNq+1k)s_{N_q+1}^k=(q,z_{N_q}^k,m_{N_q+1}^k). Appendix A provides the fixed prompt template used to serialize these inputs.

Q2: What is a feedback channel?

In Section 4.2, we distinguish feedback channels by the evaluation dimensions they score. A feedback channel denotes a set of scalar feedback values produced by the same scoring rule across candidate trajectories. The plan score riplanr_i^{\mathrm{plan}}, action score riactr_i^{\mathrm{act}}, and format score rifmtr_i^{\mathrm{fmt}} constitute three local channels. We introduce these scores in Section 4.2 and provide their operational definitions in Appendix D.2. The index i=(q,t,k)i=(q,t,k) identifies the question, step, and trajectory.

In Section 4.2 and Appendix D.1, we specify that each channel’s mean and population standard deviation are computed over candidates for the same question and step. Under the default setting δ=1\delta=1, raw scores are standardized using a denominator consisting of the standard deviation plus 10−610^{-6}. We explain in Section 4.2 and Appendix D.2 how the three component advantages are subsequently combined using weights, so each dimension is normalized before aggregation.

In Section 4.2 and Appendix D.3, we define the overall utility RiR_i by combining terminal success, local quality, and breadth signals, and normalize it within its cohort to obtain the overall advantage. All feedback is detached before policy optimization. We describe how the resulting advantages and ranking targets enter policy optimization in Section 4.3 and Appendix D.4.

AuthorsOpenReview ↗

Response to Reviewer Hjrm(3/4)

Q3: How are reward and breadth balanced during training?

In Section 4.2 and Appendix D.3, we explain how terminal success and exploration are combined through overall utility and advantage mixing. Using the default weights listed in Appendix D.1, the intermediate-turn utility is

Ri=rq,kout+0.3riloc+0.1hibreadth.R_i=r_{q,k}^{\mathrm{out}}+0.3r_i^{\mathrm{loc}}+0.1h_i^{\mathrm{breadth}}.

Here, rq,koutr_{q,k}^{\mathrm{out}} is terminal answer EM after answer-string normalization, as defined in Section 4.2 and Appendix D.3. The local reward rilocr_i^{\mathrm{loc}} combines plan, action, and format scores. We specify its construction in Appendix D.2, with raw weights of 11, 11, and 0.250.25, yielding normalized weights of 4/94/9, 4/94/9, and 1/91/9. The breadth signal hibreadthh_i^{\mathrm{breadth}} combines snapshot-use decay, evidence novelty, and uncertainty resolution, defined in Appendix D.3, with respective weights of 0.250.25, 0.50.5, and 0.250.25 listed in Appendix D.1.

We define the overall advantage AioverallA_i^{\mathrm{overall}} and local advantage AilocA_i^{\mathrm{loc}} in Section 4.2 and Appendices D.1–D.3. The overall utility is standardized within its cohort, and the intermediate-turn advantage is

Ai=0.8Aioverall+0.2Ailoc.A_i=0.8A_i^{\mathrm{overall}}+0.2A_i^{\mathrm{loc}}.

Thus, 0.80.8 specifies the overall-credit share, which already incorporates terminal, local, and breadth signals. We specify the final-turn rule in Appendix D.3, where utility consists of the terminal reward plus 0.10.1 times the final-format reward.

In Section 4.3 and Appendix D.4, we incorporate separate reward-ranking and breadth-ranking losses into the training objective. Their default coefficients, λrew=0.1\lambda_{\mathrm{rew}}=0.1 and λbr=0.05\lambda_{\mathrm{br}}=0.05, are listed in Appendix D.1. Breadth therefore influences training through both reward shaping and candidate ranking. The actual gradient contribution of each component also depends on the within-group feedback distribution, standardization, and the gradient scales of the loss terms.

Q4: How are planing, action, format scores obtained?

Appendix D.2 uses fixed text-based scoring rules. The `grounded_fraction` function lowercases the text, extracts and deduplicates terms using `\w+`, and computes the fraction of evaluated terms that occur in the source text, returning 0 for an empty evaluated set. The current observation includes the document chunk and recalled content. The plan score averages three components, namely the plan’s coverage in the combined question, observation, and previous state, its coverage in the combined observation and previous state, and the positive increase in answer coverage from the previous state to the current evidence and trajectory multiplied by evidence coverage. Answer coverage takes the maximum across training reference answers. The action score averages the predefined stem-matching result, evidence coverage in the observation, and coverage of the combined action and evidence text in the observation. Both scores lie in $[0,1]$. Intermediate-format scoring requires `thinking`, `plan`, `action`, `evidence`, `uncertainty`, and `trajectory` to appear exactly once with complete tags, in the correct order, and with nonempty content, allowing at most one complete `expand` field. The score is 1 when these conditions hold and 0 otherwise. The final-turn format score depends on whether a boxed answer string can be extracted.

Q5: How does HITRA selects the eight chunks from the 50-6400 documents?

HITRA processes its input in the order of the constructed document stream. It first concatenates and tokenizes all context documents, then divides the resulting stream into consecutive chunks of at most 5,000 tokens each. During evaluation, the number of chunks is determined by the actual context length, namely Nq=⌈∣contextq∣tok/5000⌉N_q=\left\lceil |\mathrm{context}_q|_{\mathrm{tok}}/5000\right\rceil, and each trajectory processes all NqN_q chunks sequentially. The number “8” in the paper refers to two independent settings. The training data wrapper accepts at most eight document chunks, while K=8K=8 specifies eight independently sampled candidate trajectories per question. Training contexts target approximately 32K tokens within a construction range of 30K–34K tokens, typically corresponding to six or seven chunks subject to the eight-chunk limit. For evaluation contexts containing 50–6,400 documents, the number of chunks increases dynamically with the length of the complete document stream. Each of the eight candidate trajectories maintains its own compact state and snapshot store while processing the same document-chunk sequence. After processing is complete, the system selects the final answer using each trajectory’s mean intermediate-turn log-likelihood.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.